directory-api mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Harris, Christopher P" <chris_har...@baxter.com>
Subject RE: Proper use of LdapConnectionPool
Date Tue, 27 Jan 2015 22:07:55 GMT
Hi, Emmanuel.

"Can you tell us how you do that ? Ie, are you using a plain new connection for each thread
you spawn ?"
Sure.  I can tell you how I am implementing a multi-threaded approach to read all of LDAP/AD
into memory.  I'll do the next best thing...paste my code at the end of my response.


"In any case, the TimeOut is the default LDapConnection timeout (30 seconds) :"
Yes, I noticed mention of the default timeout in your User Guide.


"You have to set the LdapConnectionConfig timeout for all the created connections to use it.
there is a setTimeout() method for that which has been added in 1.0.0-M28."
When visiting your site while seeking to explore connection pool options, I noticed that you
recently released M28 and fixed DIRAPI-217 and decided to update my pom.xml to M28 and test
out the PoolableLdapConnectionFactory.  Great job, btw.  Keep up the good work!

Oh, and your example needs to be updated to using DefaultPoolableLdapConnectionFactory instead
of PoolableLdapConnectionFactory.


"config.setTimeOut( whatever fits you );"
Very good to know.  Thank you!


"It is the right way."
Sweeeeeeet!


"Side note : you may face various problems when pulling everything from an AD server. Typically,
the AD config might not let you pull more than
1000 entries, as there is a hard limit you need to change on AD if you want to get more entries.

Otherwise, the approach - ie, using multiple threads - might seems good, but the benefit is
limited. Pulling entries from the server is fast, you should be able to get tens of thousands
per second with one single thread. I'm not sure how AD support concurrent searches anyway.
Last, not least, it's likely that AD does not allow more than a certain number of concurrent
threads to run, which might lead to contention at some point."

Ah, this is why I wanted to reach out to you guys.  You guys know this kind of in-depth information
about LDAP and AD.  So, I may adapt my code to a single-thread then.  I can live with that.
 I need to pull about 40k-60k entries, so 10's of thousands of entries per second works for
me.  I may need to run the code by you then if I go with a single-threaded approach and need
to check if I'm going about it in the most efficient manner.



And now time for some code...

import java.io.IOException;
import java.util.Iterator;
import java.util.List;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.TimeUnit;
import java.util.logging.Level;
import java.util.logging.Logger;

import org.apache.commons.pool.impl.GenericObjectPool;
import org.apache.directory.api.ldap.model.cursor.CursorException;
import org.apache.directory.api.ldap.model.cursor.SearchCursor;
import org.apache.directory.api.ldap.model.entry.Entry;
import org.apache.directory.api.ldap.model.exception.LdapException;
import org.apache.directory.api.ldap.model.message.Response;
import org.apache.directory.api.ldap.model.message.SearchRequest;
import org.apache.directory.api.ldap.model.message.SearchRequestImpl;
import org.apache.directory.api.ldap.model.message.SearchResultEntry;
import org.apache.directory.api.ldap.model.message.SearchScope;
import org.apache.directory.api.ldap.model.name.Dn;
import org.apache.directory.ldap.client.api.DefaultLdapConnectionFactory;
import org.apache.directory.ldap.client.api.LdapConnection;
import org.apache.directory.ldap.client.api.LdapConnectionConfig;
import org.apache.directory.ldap.client.api.LdapConnectionPool;
import org.apache.directory.ldap.client.api.LdapNetworkConnection;
import org.apache.directory.ldap.client.api.DefaultPoolableLdapConnectionFactory;
import org.apache.directory.ldap.client.api.ValidatingPoolableLdapConnectionFactory;
import org.apache.directory.ldap.client.api.SearchCursorImpl;
import org.apache.directory.ldap.client.template.EntryMapper;
import org.apache.directory.ldap.client.template.LdapConnectionTemplate;

/**
 * @author Chris Harris
 *
 */
public class LdapClient {
		
	public LdapClient() {
		
	}
			
	public Person searchLdapForCeo() {
		return this.searchLdapUsingHybridApproach(ceoQuery);
	}
	
	public Map<String, Person> buildLdapMap() {
		SearchCursor cursor = new SearchCursorImpl(null, 300000, TimeUnit.SECONDS);
		LdapConnection connection = new LdapNetworkConnection(host, port);
		connection.setTimeOut(300000);
		Entry entry = null;
		
		try {
			connection.bind(dn, pwd);
            			LdapClient.recursivelyGetLdapDirectReports(connection, cursor, entry, ceoQuery);
            			System.out.println("Finished all Ldap Map Builder threads...");
        		} catch (LdapException ex) {
            			Logger.getLogger(LdapClient.class.getName()).log(Level.SEVERE, null, ex);
        		} catch (CursorException ex) {
            			Logger.getLogger(LdapClient.class.getName()).log(Level.SEVERE, null, ex);
        		} finally {
            			cursor.close();
           			 try {
                			connection.close();
            			} catch (IOException ex) {
                			Logger.getLogger(LdapClient.class.getName()).log(Level.SEVERE, null, ex);
            			}
        		}
		
		return concurrentPersonMap;
	}
	
	private static Person recursivelyGetLdapDirectReports(LdapConnection connection, SearchCursor
cursor, Entry entry, String query) 
			throws CursorException {
		Person p = null;
        		EntryMapper<Person> em = Person.getEntryMapper();
        
		try {	        
	        		SearchRequest sr = new SearchRequestImpl();
	        		sr.setBase(new Dn(searchBase));
	        		StringBuilder sb = new StringBuilder(query);
	        		sr.setFilter(sb.toString());
	        		sr.setScope( SearchScope.SUBTREE );
	        		cursor = connection.search(sr);
	        		Response response;

	        		while (cursor.next() && cursor.isEntry()) {
	            			response = cursor.get();
	            			System.out.println(((SearchResultEntry)response).getEntry());
	            			entry = cursor.getEntry();
	         			
	            			// Catches entries that are disabled via the userAccountControl being set
to anything other than 512
	            			if ("512".equals(entry.get( "userAccountControl" ).getString())) {
	            				p = em.map(entry);
	            				concurrentPersonMap.put(p.getDistinguishedName(), p);
	            	
	            				if (p.getDirectReports() != null && p.getDirectReports().size() !=
0) {
	    		        			ExecutorService executor = Executors.newFixedThreadPool(p.getDirectReports().size());
	    		        
	    		        			for (String directReportDistinguishedName : p.getDirectReports()) {
	    		        				Runnable worker = new LdapPersonRetrieverRunnable(connection, cursor, entry,

	    		        				LdapDistinguishedNameUtil.buildDistinguishedNameLdapQuery(directReportDistinguishedName));
	    		        				executor.execute(worker);
	    		        			}
	    		        
	    		        			// This will make the executor accept no new threads
	    		        			// and finish all existing threads in the queue
	    		        			executor.shutdown();
	    		        			// Wait until all threads are finish
	    		        			executor.awaitTermination(Long.MAX_VALUE, TimeUnit.SECONDS);
	    	        			}
	            			} else {
	            				// Remove the Direct Report from the Manager's list of direct reports
	            				if (entry.get( "manager" ).getString() != null) {
	            					Iterator<String> it = concurrentPersonMap.get(entry.get( "manager"
).getString()).getDirectReports().iterator();
	            					while (it.hasNext()) {
	            						if ((entry.getDn().getName()).equals(it.next())) {
	            							it.remove();
	            						}
	            					}
	            				}
	            			}
	        		}
		} catch (InterruptedException ex) {
        			Logger.getLogger(LdapClient.class.getName()).log(Level.SEVERE, null, ex);
		} catch (LdapException ex) {
			System.out.println("LDAP API hung on this distinguishedName query: " + query);
			Logger.getLogger(LdapClient.class.getName()).log(Level.SEVERE, null, ex);
		}
		
		return p;
	}
	
	private static class LdapPersonRetrieverRunnable implements Runnable {
		private LdapConnection connection;
		private SearchCursor cursor;
		private Entry entry;
		private String query;
		
		public LdapPersonRetrieverRunnable(LdapConnection connection, SearchCursor cursor, Entry
entry, String query) {
			this.connection = connection;
			this.cursor = cursor;
			this.entry = entry;
			this.query = query;
		}
		
		@Override
		public void run() {
			try {
				LdapClient.recursivelyGetLdapDirectReports(connection, cursor, entry, query);
			} catch (CursorException ex) {
				Logger.getLogger(LdapClient.class.getName()).log(Level.SEVERE, null, ex);
			}
		}
	}
}


import java.util.HashMap;
import java.util.Map;
import java.util.regex.Matcher;
import java.util.regex.Pattern;

/**
 * @author Chris Harris
 *
 */
public class LdapDistinguishedNameUtil {
	
	public static String buildDistinguishedNameLdapQuery(String distinguishedName) {
		StringBuilder sb = new StringBuilder();
		sb.append("(&(objectClass=person)(distinguishedName=")
		  .append(replaceFilterSpecialCharactersWithHexRepresentation(distinguishedName))
		  .append("))");
		return sb.toString();
	}
	
	public static String replaceFilterSpecialCharactersWithHexRepresentation(String distinguishedName)
{
		Map<String, String> replacements = new HashMap<String, String>() {
			private static final long serialVersionUID = 5562696261585474518L;

		{
		    put("*", "\\\\2A");
		    put("(", "\\\\28");
		    put(")", "\\\\29");
		    put("\\", "\\\\5C");
		}};
		
		String regexp = "\\(|\\)|\\*|\\\\";
		StringBuffer sb = new StringBuffer();
		Pattern p = Pattern.compile(regexp);
		Matcher m = p.matcher(distinguishedName);

		while (m.find()) {
		    m.appendReplacement(sb, replacements.get(m.group()));
		}
		m.appendTail(sb);
		
		return sb.toString();
	}
	
	public static boolean containsIgnoreCase(String src, String what) {
	    final int length = what.length();
	    if (length == 0)
	        return true; // Empty string is contained

	    final char firstLo = Character.toLowerCase(what.charAt(0));
	    final char firstUp = Character.toUpperCase(what.charAt(0));

	    for (int i = src.length() - length; i >= 0; i--) {
	        // Quick check before calling the more expensive regionMatches() method:
	        final char ch = src.charAt(i);
	        if (ch != firstLo && ch != firstUp) {
	            continue;
	        }

	        if (src.regionMatches(true, i, what, 0, length)) {
	            return true;
	        }
	    }

	    return false;
	}
}




import java.util.ArrayList;
import java.util.Iterator;

import org.apache.directory.api.ldap.model.entry.Attribute;
import org.apache.directory.api.ldap.model.entry.Entry;
import org.apache.directory.api.ldap.model.entry.Value;
import org.apache.directory.api.ldap.model.exception.LdapException;
import org.apache.directory.ldap.client.template.EntryMapper;


/**
 * @author Chris Harris
 *
 */
public class Person {
	
	protected Person() {
		
	}
	
	private static final EntryMapper<Person> personEntryMapper = 
	new EntryMapper<Person>() {
		@Override
		public Person map( Entry entry ) throws LdapException {
			try {
				return new Person.Builder()
					.setDistinguishedName(entry.getDn().getName())
					.setFirstName(entry.get( "givenName" ).getString())
					.setLastName(entry.get( "sn" ).getString())
					.setTitle(entry.get( "title" ).getString())
					.setDepartment(entry.get( "department" ).getString())
					.setDivision(entry.get( "extensionAttribute14" ).getString())
					.setCity(entry.get( "l" ).getString())
					.setCountry(entry.get( "co" ).getString())
					.setEmail((entry.get( "mail" ) != null) ? entry.get( "mail" ).getString() : "")
					.setDeskLocation(entry.get( "postOfficeBox" ).getString())
					.setOfficePhone((entry.get( "telephoneNumber" ) != null) ? entry.get( "telephoneNumber"
).getString() : "")
					.setDirectReports(entry.get( "directreports" ))
					.build();
			} catch ( Exception e ) {
                			System.out.println( "GOTCHA: " + e.getMessage() );
                			e.printStackTrace();
                			try {
					throw( e );
				} catch (Exception e1) {
					// TODO Auto-generated catch block
					e1.printStackTrace();
				}
            			}
			return null;
		}
	};
	
	public static EntryMapper<Person> getEntryMapper() {
        		return personEntryMapper;
    	}

	public static class Builder {
		private Person person;
		
		public Builder() {
			this.person = new Person();
		}
		
		public Builder setDistinguishedName(String distinguishedName) {			
			this.person.distinguishedName = distinguishedName;
			return this;
		}
		
		public Builder setFirstName(String firstName) {
			this.person.firstName = firstName;
			return this;
		}
		
		public Builder setLastName(String lastName) {
			this.person.lastName = lastName;
			return this;
		}
		
		public Builder setDirectReports(Attribute directReports) {
			if (directReports == null) {
				this.person.directReports = null;
			} else {
				Iterator<Value<?>> it = directReports.iterator();
				ArrayList<String> dirReports = new ArrayList<String>();
	    		while (it.hasNext()) {
	    			Value<?> val = it.next();
	    			if (!LdapDistinguishedNameUtil.containsIgnoreCase(val.getString(), "Group Mailboxes")
||
	    					!LdapDistinguishedNameUtil.containsIgnoreCase(val.getString(), "Disabled Accounts"))
{
	    				dirReports.add(val.getString());
	    			}
	    		}
	    		
	    		this.person.directReports = dirReports;
			}
			return this;
		}
		
		public Builder setTitle(String title) {
			this.person.title = title;
			return this;
		}
		
		public Builder setDepartment(String department) {
			this.person.department = department;
			return this;
		}
		
		public Builder setDivision(String division) {
			this.person.division = division;
			return this;
		}
		
		public Builder setCity(String city) {
			this.person.city = city;
			return this;
		}
		
		public Builder setCountry(String country) {
			this.person.country = country;
			return this;
		}
		
		public Builder setEmail(String email) {
			this.person.email = email;
			return this;
		}
		
		public Builder setDeskLocation(String deskLocation) {
			this.person.deskLocation = deskLocation;
			return this;
		}
		
		public Builder setOfficePhone(String officePhone) {
			this.person.officePhone = officePhone;
			return this;
		}

		public Person build() {
			return this.person;
		}
	}
	
	private String distinguishedName;
	private String firstName;
	private String lastName;
	private ArrayList<String> directReports;
	private String title;
	private String department;
	private String division;
	private String city;
	private String country;
	private String email;
	private String deskLocation;
	private String officePhone;
	
	public String getDistinguishedName() {
		return distinguishedName;
	}
	public void setDistinguishedName(String distinguishedName) {
		this.distinguishedName = distinguishedName;
	}
	public String getFirstName() {
		return firstName;
	}
	public void setFirstName(String firstName) {
		this.firstName = firstName;
	}
	public String getLastName() {
		return lastName;
	}
	public void setLastName(String lastName) {
		this.lastName = lastName;
	}
	public ArrayList<String> getDirectReports() {
		return directReports;
	}
	public void setDirectReports(ArrayList<String> directReports) {
		this.directReports = directReports;
	}
	public String getTitle() {
		return title;
	}
	public void setTitle(String title) {
		this.title = title;
	}
	public String getDepartment() {
		return department;
	}
	public void setDepartment(String department) {
		this.department = department;
	}
	public String getDivision() {
		return division;
	}
	public void setDivision(String division) {
		this.division = division;
	}
	public String getCity() {
		return city;
	}
	public void setCity(String city) {
		this.city = city;
	}
	public String getCountry() {
		return country;
	}
	public void setCountry(String country) {
		this.country = country;
	}
	public String getEmail() {
		return email;
	}
	public void setEmail(String email) {
		this.email = email;
	}
	public String getDeskLocation() {
		return deskLocation;
	}
	public void setDeskLocation(String deskLocation) {
		this.deskLocation = deskLocation;
	}
	public String getOfficePhone() {
		return officePhone;
	}
	public void setOfficePhone(String officePhone) {
		this.officePhone = officePhone;
	}
}



...and finally, the relevant Junit 4.12 code snippet:
	@Test
	public void testLdapTreeBuilder() {
		long start = System.currentTimeMillis();
		Map<String, Person> ldapPersonMap = ldapClient.buildLdapMap();
		assertNotNull(ldapPersonMap);
		long end = System.currentTimeMillis();
		System.out.println("Building the Ldap map took " + Long.toString(end - start) + " MilliSeconds");
	}

 - Chris



-----Original Message-----
From: Emmanuel Lécharny [mailto:elecharny@gmail.com] 
Sent: Tuesday, January 27, 2015 2:23 AM
To: api@directory.apache.org
Subject: Re: Proper use of LdapConnectionPool

Le 27/01/15 06:11, Harris, Christopher P a écrit :
> Hi,
>
> I'm running into TimeOut issues when implementing a multi-threaded approach to read all
of LDAP/AD into memory, starting with the CEO and trickling down. 

Can you tell us how you do that ? Ie, are you using a plain new connection for each thread
you spawn ?

In any case, the TimeOut is the default LDapConnection timeout (30
seconds) :

    /** The timeout used for response we are waiting for */
    private long timeout = LdapConnectionConfig.DEFAULT_TIMEOUT;

    /** The default timeout for operation : 30 seconds */
    public static final long DEFAULT_TIMEOUT = 30000L;

That means the server is *not* responding for more than 30 seconds.

>  I haven't been using the LdapConnectionPool, so I thought that I'd give that a try to
see if it solves my TimeOut issues.  I'm setting the timeout for my cursor and connection
to 300,000 milliseconds, however the timeout consistently occurs around 30 seconds I've noticed.

You have to set the LdapConnectionConfig timeout for all the created connections to use it.
there is a setTimeout() method for that which has been added in 1.0.0-M28.
>
> Anyway, I read the docs concerning implementing an LdapConnectionPool, but the example
doesn't show you how to use the connection pool.
>
> Does the following method use the LdapConnectionPool the correct way?
>
> public Person searchLdapUsingConnectionPool() {
>              SearchCursor cursor = new SearchCursorImpl(null, 30000, TimeUnit.SECONDS);
>              LdapConnectionPool pool = null;
>              LdapConnection connection = null;
>              Person p = null;
>              try {
>                     LdapConnectionConfig config = new LdapConnectionConfig();
>                     config.setLdapHost( host );
>                     config.setLdapPort( port );
>                     config.setName( dn );
>                     config.setCredentials( pwd );
>                     DefaultPoolableLdapConnectionFactory factory = new DefaultPoolableLdapConnectionFactory(
config );
>                     pool = new LdapConnectionPool( factory );
>                     pool.setTestOnBorrow( true );
>                     connection = pool.getConnection();

It's all good. You can now change the config timeout :
config.setTimeOut( whatever fits you );
>
>                     Entry entry = null;
>
>
>             SearchRequest sr = new SearchRequestImpl();
>             sr.setBase(new Dn(searchBase));
>             StringBuilder sb = new StringBuilder(ceoQuery);
>             sr.setFilter(sb.toString());
>             sr.setScope( SearchScope.SUBTREE );
>             cursor = connection.search(sr);
>             Response response;
>
>             while (cursor.next() && cursor.isEntry()) {
>                 response = cursor.get();
>                 System.out.println(((SearchResultEntry)response).getEntry());
>                 entry = cursor.getEntry();
>                 EntryMapper<Person> em = Person.getEntryMapper();
>                    p = em.map(entry);
>             }
>         } catch (LdapException ex) {
>             Logger.getLogger(LdapClient.class.getName()).log(Level.SEVERE, null, ex);
>         } catch (CursorException ex) {
>             Logger.getLogger(LdapClient.class.getName()).log(Level.SEVERE, null, ex);
>         } finally {
>             cursor.close();
>             try {
>                 pool.releaseConnection(connection);
>             } catch (LdapException ex) {
>             Logger.getLogger(LdapClient.class.getName()).log(Level.SEVERE, null, ex);
>                     }
>         }
>
>         return p;
>        }
>
> It seems like I just need to grab a connection via pool.getConnection(), but I don't
know for sure if I'm using it the recommended way.  

It is the right way.


Side note : you may face various problems when pulling everything from an AD server. Typically,
the AD config might not let you pull more than
1000 entries, as there is a hard limit you need to change on AD if you want to get more entries.

Otherwise, the approach - ie, using multiple threads - might seems good, but the benefit is
limited. Pulling entries from the server is fast, you should be able to get tens of thousands
per second with one single thread. I'm not sure how AD support concurrent searches anyway.
Last, not least, it's likely that AD does not allow more than a certain number of concurrent
threads to run, which might lead to contention at some point.

Hope it helps.

Emmanuel

The information transmitted is intended only for the person(s) or entity to which it is addressed
and may contain confidential and/or legally privileged material. Delivery of this message
to any person other than the intended recipient(s) is not intended in any way to waive privilege
or confidentiality. Any review, retransmission, dissemination or other use of, or taking of
any action in reliance upon, this information by entities other than the intended recipient
is prohibited. If you receive this in error, please contact the sender and delete the material
from any computer.

For Translation:

http://www.baxter.com/email_disclaimer
Mime
View raw message