[Swan] racoon ipsec with openswan

[Ted Toth]

I'm trying to get racoon (RHEL5) to talk to openswan (RHEL6) but I'm
not having any success.

Racoon

racoon.conf :
path include "/etc/racoon";
path pre_shared_key "/etc/racoon/psk.txt";
path certificate "/etc/racoon/certs";
path script "/etc/racoon/scripts";

sainfo anonymous
{
    pfs_group 2;
    lifetime time 36 hours ;
    encryption_algorithm 3des, aes ;
    authentication_algorithm hmac_sha1 ;
    compression_algorithm deflate ;
}
#base
remote anonymous
{
    exchange_mode main;
    lifetime time 36 hours ;
    proposal {
        encryption_algorithm 3des;
        hash_algorithm sha1;
        authentication_method pre_shared_key;
        dh_group 2;
    }
}

key.conf:
#!/sbin/setkey -f
spdflush;

spdadd 192.168.10.200 192.168.10.10 any -P out ipsec esp/transport//require;
spdadd 192.168.10.10 192.168.10.200 any -P in  ipsec esp/transport//require;

psk.txt
192.168.10.10        123456789


Openswan
c200.conf:
conn c200
     auto=start
     authby=secret
     type=transport
     left=192.168.10.10
     right=192.168.10.200
     ikelifetime=24h
     salifetime=24h

c200.secrets:
192.168.10.10 192.168.10.200 : PSK "123456789"

ipsec.conf
#
# Manual:     ipsec.conf.5
#
# Please place your own config files in /etc/ipsec.d/ ending in .conf

version    2.0    # conforms to second version of ipsec.conf specification

# basic configuration
config setup
    # Debug-logging controls:  "none" for (almost) none, "all" for lots.
    # klipsdebug=none
    # plutodebug="control parsing"
    # For Red Hat Enterprise Linux and Fedora, leave protostack=netkey
    protostack=netkey
    nat_traversal=yes
    virtual_private=
    oe=off
    # Enable this if you see "failed to find any available worker"
    # nhelpers=0

#You may put your configuration (.conf) file in the "/etc/ipsec.d/"
and uncomment this.
include /etc/ipsec.d/*.conf

Logging from racoon side:
Dec 11 18:16:16 comms racoon: INFO: respond new phase 1 negotiation:
192.168.10.200[500]<=>192.168.10.10[500]
Dec 11 18:16:16 comms racoon: INFO: begin Identity Protection mode.
Dec 11 18:16:16 comms racoon: INFO: received Vendor ID: DPD
Dec 11 18:16:16 comms racoon: INFO: received Vendor ID: RFC 3947
Dec 11 18:16:16 comms racoon: INFO: received Vendor ID:
draft-ietf-ipsec-nat-t-ike-03
Dec 11 18:16:16 comms racoon: INFO: received Vendor ID:
draft-ietf-ipsec-nat-t-ike-02#012
Dec 11 18:16:16 comms racoon: INFO: received Vendor ID:
draft-ietf-ipsec-nat-t-ike-02
Dec 11 18:16:16 comms racoon: INFO: received Vendor ID:
draft-ietf-ipsec-nat-t-ike-00
Dec 11 18:17:06 comms racoon: ERROR: phase1 negotiation failed due to
time up. 7b6f84e2992d11b6:64a1d35251dea3c0

Logging from openswan side:
Dec 11 17:08:38 comms pluto[2054]: packet from 192.168.10.200:500:
phase 1 message is part of an unknown exchange
Dec 11 17:08:48 comms pluto[2054]: packet from 192.168.10.200:500:
phase 1 message is part of an unknown exchange
Dec 11 17:08:50 comms pluto[2054]: "c200" #20: initiating Main Mode
Dec 11 17:08:50 comms pluto[2054]: "c200" #20: received Vendor ID
payload [Dead Peer Detection]
Dec 11 17:08:50 comms pluto[2054]: "c200" #20: transition from state
STATE_MAIN_I1 to state STATE_MAIN_I2
Dec 11 17:08:50 comms pluto[2054]: "c200" #20: STATE_MAIN_I2: sent
MI2, expecting MR2
Dec 11 17:08:50 comms pluto[2054]: "c200" #20: I will NOT send an
initial contact payload
Dec 11 17:08:50 comms pluto[2054]: "c200" #20: Not sending INITIAL_CONTACT
Dec 11 17:08:50 comms pluto[2054]: "c200" #20: transition from state
STATE_MAIN_I2 to state STATE_MAIN_I3
Dec 11 17:08:50 comms pluto[2054]: "c200" #20: STATE_MAIN_I3: sent
MI3, expecting MR3
Dec 11 17:08:59 comms pluto[2054]: packet from 192.168.10.200:500:
phase 1 message is part of an unknown exchange
Dec 11 17:09:01 comms pluto[2054]: "c200" #20: discarding duplicate
packet; already STATE_MAIN_I3
Dec 11 17:09:11 comms pluto[2054]: "c200" #20: discarding duplicate
packet; already STATE_MAIN_I3
Dec 11 17:09:21 comms pluto[2054]: "c200" #20: discarding duplicate
packet; already STATE_MAIN_I3
Dec 11 17:09:31 comms pluto[2054]: "c200" #20: discarding duplicate
packet; already STATE_MAIN_I3
Dec 11 17:10:00 comms pluto[2054]: "c200" #20: max number of
retransmissions (2) reached STATE_MAIN_I3.  Possible authentication
failure: no acceptable response to our first encrypted message
Dec 11 17:10:00 comms pluto[2054]: "c200" #20: starting keying attempt
2 of an unlimited number
Dec 11 17:10:00 comms pluto[2054]: "c200" #21: initiating Main Mode to
replace #20
Dec 11 17:10:00 comms pluto[2054]: "c200" #21: received Vendor ID
payload [Dead Peer Detection]
Dec 11 17:10:00 comms pluto[2054]: "c200" #21: transition from state
STATE_MAIN_I1 to state STATE_MAIN_I2
Dec 11 17:10:00 comms pluto[2054]: "c200" #21: STATE_MAIN_I2: sent
MI2, expecting MR2
Dec 11 17:10:00 comms pluto[2054]: "c200" #21: I will NOT send an
initial contact payload
Dec 11 17:10:00 comms pluto[2054]: "c200" #21: Not sending INITIAL_CONTACT
Dec 11 17:10:00 comms pluto[2054]: "c200" #21: transition from state
STATE_MAIN_I2 to state STATE_MAIN_I3
Dec 11 17:10:00 comms pluto[2054]: "c200" #21: STATE_MAIN_I3: sent
MI3, expecting MR3
Dec 11 17:10:10 comms pluto[2054]: "c200" #21: discarding duplicate
packet; already STATE_MAIN_I3
Dec 11 17:10:20 comms pluto[2054]: "c200" #21: discarding duplicate
packet; already STATE_MAIN_I3


It's difficult for me a mere mortal to parse the logs and figure out
what the issue is, any ideas? Anyone successfully done this, if so can
you share your config?

Ted