# HG changeset patch # User Sebastien Decugis # Date 1297242534 -32400 # Node ID e387d5c6b6f5c73f3436bb2f0400fc78e69b27f4 # Parent 4ffbc9f1e9229d1f399462d5f7457e37852fb853 Added support for Internationalized Domain Names (IDNA) using GNU libidn diff -r 4ffbc9f1e922 -r e387d5c6b6f5 INSTALL.Fedora --- a/INSTALL.Fedora Wed Feb 09 15:26:58 2011 +0900 +++ b/INSTALL.Fedora Wed Feb 09 18:08:54 2011 +0900 @@ -3,7 +3,7 @@ Dependencies on Fedora 13 (from minimal system): -# yum install cmake make gcc flex bison lksctp-tools-devel gnutls-devel libgcrypt-devel +# yum install cmake make gcc flex bison lksctp-tools-devel gnutls-devel libgcrypt-devel libidn-devel In addition, if you have not already retrieved the latest source: # yum install mercurial diff -r 4ffbc9f1e922 -r e387d5c6b6f5 INSTALL.FreeBSD --- a/INSTALL.FreeBSD Wed Feb 09 15:26:58 2011 +0900 +++ b/INSTALL.FreeBSD Wed Feb 09 18:08:54 2011 +0900 @@ -19,7 +19,7 @@ Install minimal system + ports using initial installer /usr/sbin/sysinstall -2) Install cmake +2) Install 'cmake' a) from sources: # cd /usr/ports/devel/cmake @@ -29,12 +29,12 @@ # pkg_add -v -r cmake -3) Install mercurial (optional) - (replace "cmake" by "mercurial" in the previous command) +3) Install 'mercurial' (optional) + (replace 'cmake' by 'mercurial' in the previous command) -4) Install flex and bison, same way. +4) Install 'flex' and 'bison', same way. -5) Install gnutls, same way also. +5) Install 'gnutls' and 'libidn', same way also. 6) Retrieve freeDiameter source code: # cd ~ diff -r 4ffbc9f1e922 -r e387d5c6b6f5 INSTALL.OpenSUSE --- a/INSTALL.OpenSUSE Wed Feb 09 15:26:58 2011 +0900 +++ b/INSTALL.OpenSUSE Wed Feb 09 18:08:54 2011 +0900 @@ -3,7 +3,7 @@ Dependencies on OpenSUSE 11.3 (from minimal server system installation): -# zypper install cmake make gcc flex bison lksctp-tools-devel libgnutls-devel libgcrypt-devel +# zypper install cmake make gcc flex bison lksctp-tools-devel libgnutls-devel libgcrypt-devel libidn-devel # zypper install mercurial Following dependencies are optional, depending on which extensions you plan to compile diff -r 4ffbc9f1e922 -r e387d5c6b6f5 INSTALL.Ubuntu --- a/INSTALL.Ubuntu Wed Feb 09 15:26:58 2011 +0900 +++ b/INSTALL.Ubuntu Wed Feb 09 18:08:54 2011 +0900 @@ -17,7 +17,9 @@ ============================================ The following packages are required to compile freeDiameter from source: - cmake make gcc flex bison libsctp1 libsctp-dev libgnutls-dev libgcrypt-dev + cmake make gcc flex bison libsctp1 libsctp-dev libgnutls-dev libgcrypt-dev libidn11-dev + +(note that libidn and libsctp can be avoided by defining DISABLE_SCTP and DIAMID_IDNA_REJECT) Additionnaly, these ones may be useful: mercurial gdb @@ -42,7 +44,7 @@ the following commands should generate the freeDiameter packages for you: # Install the dependencies for building the source: -sudo apt-get -y install mercurial cmake make gcc bison flex libsctp-dev libgnutls-dev libgcrypt-dev ssl-cert debhelper fakeroot \ +sudo apt-get -y install mercurial cmake make gcc bison flex libsctp-dev libgnutls-dev libgcrypt-dev libidn11-dev ssl-cert debhelper fakeroot \ libpq-dev libmysqlclient-dev libxml2-dev swig python-dev # Retrieve the latest version of the source package @@ -83,7 +85,7 @@ Step by step instructions without using the debhelper tools: 1) Install all packages dependencies -# sudo apt-get install mercurial cmake make gcc bison flex libsctp-dev libgnutls-dev libgcrypt-dev +# sudo apt-get install mercurial cmake make gcc bison flex libsctp-dev libgnutls-dev libgcrypt-dev libidn11-dev 2) (OPTION) If you will compile modules that require postgresql, also install: # sudo apt-get install libpq-dev diff -r 4ffbc9f1e922 -r e387d5c6b6f5 cmake/Modules/FindIDNA.cmake --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/cmake/Modules/FindIDNA.cmake Wed Feb 09 18:08:54 2011 +0900 @@ -0,0 +1,40 @@ +# - Try to find GNU IDN library and headers +# Once done, this will define +# +# IDNA_FOUND - system has IDNA +# IDNA_INCLUDE_DIR - the IDNA include directories () +# IDNA_LIBRARIES - link these to use IDNA (idna_to_ascii_8z) + +if (IDNA_INCLUDE_DIR AND IDNA_LIBRARIES) + set(IDNA_FIND_QUIETLY TRUE) +endif (IDNA_INCLUDE_DIR AND IDNA_LIBRARIES) + +# Include dir +find_path(IDNA_INCLUDE_DIR + NAMES idna.h +) + +# Library +find_library(IDNA_LIBRARY + NAMES idn +) + + +# handle the QUIETLY and REQUIRED arguments and set IDNA_FOUND to TRUE if +# all listed variables are TRUE +INCLUDE(FindPackageHandleStandardArgs) +FIND_PACKAGE_HANDLE_STANDARD_ARGS(IDNA DEFAULT_MSG IDNA_LIBRARY IDNA_INCLUDE_DIR) + +# If we successfully found the idn library then add the library to the +# IDNA_LIBRARIES cmake variable otherwise set IDNA_LIBRARIES to nothing. +IF(IDNA_FOUND) + SET( IDNA_LIBRARIES ${IDNA_LIBRARY} ) +ELSE(IDNA_FOUND) + SET( IDNA_LIBRARIES ) +ENDIF(IDNA_FOUND) + + +# Lastly make it so that the IDNA_LIBRARY and IDNA_INCLUDE_DIR variables +# only show up under the advanced options in the gui cmake applications. +MARK_AS_ADVANCED( IDNA_LIBRARY IDNA_INCLUDE_DIR ) + diff -r 4ffbc9f1e922 -r e387d5c6b6f5 contrib/OpenWRT/packages/freeDiameter/Makefile --- a/contrib/OpenWRT/packages/freeDiameter/Makefile Wed Feb 09 15:26:58 2011 +0900 +++ b/contrib/OpenWRT/packages/freeDiameter/Makefile Wed Feb 09 18:08:54 2011 +0900 @@ -75,6 +75,7 @@ cmake \ -DCMAKE_PREFIX_PATH:PATH=$(STAGING_DIR)/usr \ -DCMAKE_INSTALL_PREFIX:PATH=/usr \ + -DDIAMID_IDNA_REJECT:BOOL=ON \ -DBUILD_TESTING:BOOL=OFF \ -DCMAKE_BUILD_TYPE:STRING=DebianPackage \ -DDEFAULT_CONF_PATH:PATH=/etc/freeDiameter \ diff -r 4ffbc9f1e922 -r e387d5c6b6f5 contrib/debian/control --- a/contrib/debian/control Wed Feb 09 15:26:58 2011 +0900 +++ b/contrib/debian/control Wed Feb 09 18:08:54 2011 +0900 @@ -4,7 +4,7 @@ Maintainer: Sebastien Decugis Build-Depends: debhelper ( >= 7.3.9), cmake, make, gcc, bison, flex, - libsctp-dev, libgnutls-dev, libgcrypt-dev, + libsctp-dev, libgnutls-dev, libgcrypt-dev, libidn11-dev, libpq-dev, libmysqlclient-dev, libxml2-dev, swig, python-dev Standards-Version: 3.8.3 Homepage: http://www.freediameter.net diff -r 4ffbc9f1e922 -r e387d5c6b6f5 contrib/nightly_tests/idnaignore.conf --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/contrib/nightly_tests/idnaignore.conf Wed Feb 09 18:08:54 2011 +0900 @@ -0,0 +1,4 @@ + +set(CTEST_BUILD_OPTIONS "${CTEST_BUILD_OPTIONS} -DDIAMID_IDNA_IGNORE:BOOL=ON") + +set(CTEST_BUILD_NAME "IDNA Ignore") diff -r 4ffbc9f1e922 -r e387d5c6b6f5 contrib/nightly_tests/idnareject.conf --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/contrib/nightly_tests/idnareject.conf Wed Feb 09 18:08:54 2011 +0900 @@ -0,0 +1,4 @@ + +set(CTEST_BUILD_OPTIONS "${CTEST_BUILD_OPTIONS} -DDIAMID_IDNA_REJECT:BOOL=ON") + +set(CTEST_BUILD_NAME "IDNA Reject") diff -r 4ffbc9f1e922 -r e387d5c6b6f5 contrib/nightly_tests/tests.list --- a/contrib/nightly_tests/tests.list Wed Feb 09 15:26:58 2011 +0900 +++ b/contrib/nightly_tests/tests.list Wed Feb 09 18:08:54 2011 +0900 @@ -1,4 +1,6 @@ allext alldefault nosctp +idnaignore +idnareject #noext diff -r 4ffbc9f1e922 -r e387d5c6b6f5 include/freeDiameter/CMakeLists.txt --- a/include/freeDiameter/CMakeLists.txt Wed Feb 09 15:26:58 2011 +0900 +++ b/include/freeDiameter/CMakeLists.txt Wed Feb 09 18:08:54 2011 +0900 @@ -18,8 +18,13 @@ # Create the absolute path for searching extensions SET(DEFAULT_EXTENSIONS_PATH ${CMAKE_INSTALL_PREFIX}/${INSTALL_EXTENSIONS_SUFFIX}) +# IDNA considerations +OPTION(DIAMID_IDNA_IGNORE "Ignore completly invalid characters in Diameter Identities (process blindly)?" OFF) +IF (NOT DIAMID_IDNA_IGNORE) + OPTION (DIAMID_IDNA_REJECT "Reject internationalized Diameter Identities, do not attempt to convert it (stringprep) ?" OFF) +ENDIF (NOT DIAMID_IDNA_IGNORE) -MARK_AS_ADVANCED(DISABLE_SCTP DEBUG_SCTP SCTP_USE_MAPPED_ADDRESSES ERRORS_ON_TODO) +MARK_AS_ADVANCED(DISABLE_SCTP DEBUG_SCTP SCTP_USE_MAPPED_ADDRESSES ERRORS_ON_TODO DIAMID_IDNA_IGNORE DIAMID_IDNA_REJECT) ######################## ### System checks part @@ -108,6 +113,27 @@ SET(SCTP_INCLUDE_DIR ${SCTP_INCLUDE_DIR} PARENT_SCOPE) SET(SCTP_LIBRARIES ${SCTP_LIBRARIES} PARENT_SCOPE) +# IDNA process: we use libidn from GNU (unless the function & header files are included in libc) +IF(NOT DIAMID_IDNA_IGNORE AND NOT DIAMID_IDNA_REJECT) + FIND_PACKAGE(IDNA) + SET(CHECK_IDNA_SOURCE_CODE " + #include + int main() { + return idna_to_ascii_8z(NULL, NULL, 0); + } + ") + SET(CMAKE_REQUIRED_INCLUDES ${IDNA_INCLUDE_DIR}) + SET(CMAKE_REQUIRED_LIBRARIES ${IDNA_LIBRARIES}) + CHECK_C_SOURCE_COMPILES("${CHECK_IDNA_SOURCE_CODE}" HAS_IDNA_SUPPORT) + IF(NOT HAS_IDNA_SUPPORT) + MESSAGE(SEND_ERROR "Unable to find idna.h header or idna_to_ascii_8z function, please install libidn-dev or equivalent, or set DIAMID_IDNA_IGNORE or DIAMID_IDNA_REJECT") + ENDIF(NOT HAS_IDNA_SUPPORT) +ELSE (NOT DIAMID_IDNA_IGNORE AND NOT DIAMID_IDNA_REJECT) + MESSAGE(STATUS "Non-default Internationalized Domain Names (IDN) behavior selected (no stringprep).") +ENDIF(NOT DIAMID_IDNA_IGNORE AND NOT DIAMID_IDNA_REJECT) +SET(IDNA_INCLUDE_DIR ${IDNA_INCLUDE_DIR} PARENT_SCOPE) +SET(IDNA_LIBRARIES ${IDNA_LIBRARIES} PARENT_SCOPE) + # Require GNU TLS for building the library FIND_PACKAGE(GnuTLS REQUIRED) @@ -134,7 +160,9 @@ ########################## # LFDPROTO_LIBS = libraries required by the libfdproto. -SET(LFDPROTO_LIBS ${CLOCK_GETTIME_LIBS} ${CMAKE_THREAD_LIBS_INIT} PARENT_SCOPE) +SET(LFDPROTO_LIBS ${CLOCK_GETTIME_LIBS} ${CMAKE_THREAD_LIBS_INIT} ${IDNA_LIBRARIES} PARENT_SCOPE) +# And includes paths +SET(LFDPROTO_INCLUDES ${IDNA_INCLUDE_DIR} PARENT_SCOPE) # Dependencies: the libraries required by any code linking to libfdproto. SET(LFDPROTO_LINK_INTERFACES ${CMAKE_THREAD_LIBS_INIT} PARENT_SCOPE) diff -r 4ffbc9f1e922 -r e387d5c6b6f5 include/freeDiameter/freeDiameter-host.h.in --- a/include/freeDiameter/freeDiameter-host.h.in Wed Feb 09 15:26:58 2011 +0900 +++ b/include/freeDiameter/freeDiameter-host.h.in Wed Feb 09 18:08:54 2011 +0900 @@ -48,6 +48,8 @@ #cmakedefine SCTP_USE_MAPPED_ADDRESSES #cmakedefine SCTP_CONNECTX_4_ARGS #cmakedefine SKIP_DLCLOSE +#cmakedefine DIAMID_IDNA_IGNORE +#cmakedefine DIAMID_IDNA_REJECT #cmakedefine ERRORS_ON_TODO #cmakedefine DEBUG diff -r 4ffbc9f1e922 -r e387d5c6b6f5 libfdproto/CMakeLists.txt --- a/libfdproto/CMakeLists.txt Wed Feb 09 15:26:58 2011 +0900 +++ b/libfdproto/CMakeLists.txt Wed Feb 09 18:08:54 2011 +0900 @@ -20,6 +20,9 @@ # Save the list of files for testcases in the core's directory SET(LFDPROTO_SRC ${LFDPROTO_SRC} PARENT_SCOPE) +# Include path +INCLUDE_DIRECTORIES(${LFDPROTO_INCLUDES}) + # Build as a shared library ADD_LIBRARY(libfdproto SHARED ${LFDPROTO_SRC}) diff -r 4ffbc9f1e922 -r e387d5c6b6f5 libfdproto/ostr.c --- a/libfdproto/ostr.c Wed Feb 09 15:26:58 2011 +0900 +++ b/libfdproto/ostr.c Wed Feb 09 18:08:54 2011 +0900 @@ -35,6 +35,11 @@ #include "fdproto-internal.h" +#if (!defined(DIAMID_IDNA_IGNORE) && !defined(DIAMID_IDNA_REJECT)) +/* Process IDNA with stringprep -- See RFC5890 -- and libidn documentation... */ +#include /* idna_to_ascii_8z() */ +#endif /* !defined(DIAMID_IDNA_IGNORE) && !defined(DIAMID_IDNA_REJECT) */ + /* Similar to strdup with (must be verified) os0_t */ os0_t os0dup_int(os0_t s, size_t l) { os0_t r; @@ -88,9 +93,15 @@ /* Check if the string contains only ASCII */ int fd_os_is_valid_DiameterIdentity(uint8_t * os, size_t ossz) { +#ifdef DIAMID_IDNA_IGNORE + + /* Allow anything */ + +#else /* DIAMID_IDNA_IGNORE */ + int i; - /* Allow letters, digits, hyphen, dot */ + /* Allow only letters, digits, hyphen, dot */ for (i=0; i < ossz; i++) { if (os[i] > 'z') break; @@ -105,9 +116,53 @@ break; } if (i < ossz) { - TRACE_DEBUG(INFO, "Invalid character '%c' in DiameterIdentity '%.*s'", os[i], ossz, os); + int nb = 1; + /* To get a better display, check if the invalid char is UTF-8 */ + if ((os[i] & 0xE0) == 0xC0 /* 110xxxxx */) { + if ((i < ossz - 1) && ((os[i + 1] & 0xC0) == 0x80 /* 10xxxxxx */)) + nb = 2; + goto disp; + } + if ((os[i] & 0xF0) == 0xE0 /* 1110xxxx */) { + if ((i < ossz - 2) && ((os[i + 1] & 0xC0) == 0x80 /* 10xxxxxx */) + && ((os[i + 2] & 0xC0) == 0x80 /* 10xxxxxx */)) + nb = 3; + goto disp; + } + if ((os[i] & 0xF8) == 0xF0 /* 11110xxx */) { + if ((i < ossz - 3) && ((os[i + 1] & 0xC0) == 0x80 /* 10xxxxxx */) + && ((os[i + 2] & 0xC0) == 0x80 /* 10xxxxxx */) + && ((os[i + 3] & 0xC0) == 0x80 /* 10xxxxxx */)) + nb = 4; + goto disp; + } + if ((os[i] & 0xFC) == 0xF8 /* 111110xx */) { + if ((i < ossz - 4) && ((os[i + 1] & 0xC0) == 0x80 /* 10xxxxxx */) + && ((os[i + 2] & 0xC0) == 0x80 /* 10xxxxxx */) + && ((os[i + 3] & 0xC0) == 0x80 /* 10xxxxxx */) + && ((os[i + 4] & 0xC0) == 0x80 /* 10xxxxxx */)) + nb = 5; + goto disp; + } + if ((os[i] & 0xFE) == 0xFC /* 1111110x */) { + if ((i < ossz - 5) && ((os[i + 1] & 0xC0) == 0x80 /* 10xxxxxx */) + && ((os[i + 2] & 0xC0) == 0x80 /* 10xxxxxx */) + && ((os[i + 3] & 0xC0) == 0x80 /* 10xxxxxx */) + && ((os[i + 4] & 0xC0) == 0x80 /* 10xxxxxx */) + && ((os[i + 5] & 0xC0) == 0x80 /* 10xxxxxx */)) + nb = 6; + goto disp; + } + /* otherwise, we just display the hex code */ + TRACE_DEBUG(INFO, "Invalid character (0xhhX) at offset %d in DiameterIdentity '%.*s'", os[i], i+1, ossz, os); + return 0; +disp: + TRACE_DEBUG(INFO, "Invalid character '%.*s' at offset %d in DiameterIdentity '%.*s'", nb, os + i, i+1, ossz, os); return 0; } + +#endif /* DIAMID_IDNA_IGNORE */ + return 1; } @@ -118,16 +173,38 @@ *outsz = strlen(*id); +#ifndef DIAMID_IDNA_IGNORE + if (!fd_os_is_valid_DiameterIdentity((os0_t)*id, *outsz)) { - char buf[HOST_NAME_MAX]; + +#ifdef DIAMID_IDNA_REJECT + + TRACE_DEBUG(INFO, "The string '%s' is not a valid DiameterIdentity!", *id); + TRACE_DEBUG(INFO, "Returning EINVAL since fD is compiled with option DIAMID_IDNA_REJECT."); + return EINVAL; + +#else /* DIAMID_IDNA_REJECT */ + + char *processed; + int ret; - TODO("Stringprep in into buf"); - TRACE_DEBUG(INFO, "The string '%s' is not a valid DiameterIdentity, it was changed to '%s'", *id, buf); - TODO("Realloc *id if !memory"); - /* copy buf */ - /* update the size */ - return ENOTSUP; - } else { + ret = idna_to_ascii_8z ( *id, &processed, IDNA_USE_STD3_ASCII_RULES ); + if (ret == IDNA_SUCCESS) { + TRACE_DEBUG(INFO, "The string '%s' is not a valid DiameterIdentity, it was changed to '%s'", *id, processed); + if (memory == 0) + free(*id); + *id = processed; + *outsz = strlen(processed); + /* Done! */ + } else { + TRACE_DEBUG(INFO, "The string '%s' is not a valid DiameterIdentity and cannot be sanitanized: %s", *id, idna_strerror (ret)); + return EINVAL; + } + +#endif /* DIAMID_IDNA_REJECT */ + } else +#endif /* ! DIAMID_IDNA_IGNORE */ + { if (memory == 1) { CHECK_MALLOC( *id = os0dup(*id, *outsz) ); } @@ -137,8 +214,6 @@ - - /********************************************************************************************************/ /* Hash function -- credits to Austin Appleby, thank you ^^ */ /* See http://murmurhash.googlepages.com for more information on this function */ diff -r 4ffbc9f1e922 -r e387d5c6b6f5 tests/testostr.c --- a/tests/testostr.c Wed Feb 09 15:26:58 2011 +0900 +++ b/tests/testostr.c Wed Feb 09 18:08:54 2011 +0900 @@ -37,6 +37,10 @@ #define TEST_STR (os0_t)"This is my test string (with extra unused data)" +/* The following string contains UTF-8 encoded characters (Chinese characters) */ +#define TEST_IDN_UTF8 "freeDiameter.中国" +#define TEST_IDN_CONV "freeDiameter.xn--fiqs8s" + /* Main test routine */ int main(int argc, char *argv[]) { @@ -66,7 +70,62 @@ memcpy(buf + 1, TEST_STR, CONSTSTRLEN(TEST_STR)); CHECK( hash, fd_os_hash(buf + 1, CONSTSTRLEN(TEST_STR)) ); } + + /* Check the Diameter Identity functions */ + { + char * res; + size_t len; + + /* A valid ASCII domain name */ + res = TEST_IDN_CONV; + CHECK( 0, fd_os_validate_DiameterIdentity(&res, &len, 1) ); + CHECK( 0, strcasecmp(res, TEST_IDN_CONV) ); /* the function does not change a valid DN */ + CHECK( 0, fd_os_validate_DiameterIdentity(&res, &len, 0) ); + CHECK( 0, strcasecmp(res, TEST_IDN_CONV) ); + CHECK( CONSTSTRLEN(TEST_IDN_CONV), len ); + free(res); + + /* Now, an invalid string */ + res = TEST_IDN_UTF8; + + #ifdef DIAMID_IDNA_IGNORE + + /* The UTF-8 chars are considered valid */ + CHECK( 1, fd_os_is_valid_DiameterIdentity((os0_t)TEST_IDN_UTF8, CONSTSTRLEN(TEST_IDN_UTF8) ); + + /* The string should be passed unmodified */ + CHECK( 0, fd_os_validate_DiameterIdentity(&res, &len, 1) ); + CHECK( 0, strcasecmp(res, TEST_IDN_UTF8) ); + CHECK( 0, fd_os_cmp(res, len, TEST_IDN_UTF8, CONSTSTRLEN(TEST_IDN_UTF8)) ); + CHECK( 0, fd_os_almostcasecmp(res, len, TEST_IDN_UTF8, CONSTSTRLEN(TEST_IDN_UTF8)) ); + CHECK( 0, fd_os_validate_DiameterIdentity(&res, &len, 0) ); + CHECK( 0, strcasecmp(res, TEST_IDN_UTF8) ); + CHECK( CONSTSTRLEN(TEST_IDN_UTF8), len ); + free(res); + + #else /* DIAMID_IDNA_IGNORE */ + + /* The UTF-8 chars are recognized as invalid DiameterIdentity */ + CHECK( 0, fd_os_is_valid_DiameterIdentity((os0_t)TEST_IDN_UTF8, CONSTSTRLEN(TEST_IDN_UTF8) )); + + # ifdef DIAMID_IDNA_REJECT + + /* The string must be rejected */ + CHECK( EINVAL, fd_os_validate_DiameterIdentity(&res, &len, 1) ); + + # else /* DIAMID_IDNA_REJECT */ + + /* The string should be transformed into TEST_IDN_CONV */ + CHECK( 0, fd_os_validate_DiameterIdentity(&res, &len, 1) ); + CHECK( 0, strcasecmp(res, TEST_IDN_CONV) ); + CHECK( CONSTSTRLEN(TEST_IDN_CONV), len ); + free(res); + + # endif /* DIAMID_IDNA_REJECT */ + #endif /* DIAMID_IDNA_IGNORE */ + } + /* That's all for the tests yet */ PASSTEST(); }