hi,
I am some proposed idea for discussion. Some of them are design principles
or concept, and some are more concrete points about design on specific
items. The points are as follows:
* - indicate items that I consider as major changes
**
1. Use of Java Package
- do not use a org.apache.droids.api package. Just put the API
interface in individual package.
- do not use *.impl unless there are 3 or more. (i don't really have
strong opinion at this point, just don't like to have some impl
package with
just one or two class(es))
2. General coding practice/standard
- use protected instead of private by default. This will allow users
to extend and replace any of our class easier.
- do not use final for LinkTask/Link. I have a use case that I want to
extend Link/LinkTask as a JPA Entity Bean. And JPA doesn't allow a class
without default constructor, i.e. we can't final the URL field. For the
classes that are not expected to be subclassed by users, it's ok to use
final.
- Use Java standard interface rather than introducing our own
interface unless there are obvious value-added. e.g. replace
TaskQueue with
java.util.Queue<Task>
- *Use Spring for droids-core
- i.e. allow droids-core depends on Spring
- a droid needs a whole object graph to work. as a framework, we
want different components be configurable. It's better to
rely on an IoC
framework to manage the dependency and configuration. Spring
is the most
popular IoC framework. it will also make testing much easier.
and user needs
not to change code (but to change xml) if they want to change certain
behavior of our core classes.
- there are some special benefits of using spring, e.g. it supports
annotation and autowiring. Take an example, I could define a
field like
"@Autowired Collection<Filter> filters", and when I add a new
filter, then,
i create 2 classes like "@Component public class XXXFilter",
and in runtime,
the filters field will be injected with a collection of the 2
classes. It
makes development and configuration real simple. (and there
are also ways to
change the autowiring behavior)
- Spring facilitates the use of http remoting. And it is easy to
replace implementation class to do remoting or other interception.
3. Specific concept/component. Some of the points in this section
are just my comment to the concept, but not any proposal for action.
- Droids/Crawler
- Our top level concept. We only use Droid but not Crawler. I use
the generic term Crawler in this message.
- Link, LinkTask, Task
- Task is a valid concept. A task is the unit that work by the worker.
A link refer to the link only. I have no objection to this
concept. But in
implementation, it seems there is no much need to implement the Task
concept. (naming an interface as "nextTask" is ok but it
seems no need to
have a class or interface called "Task")
- a crawler works with links.and we don't normally non-link related
task that goes beyond the scope of a crawler framework.
- See the Link-centric design bullet for more info about link
- Fetcher
- we do not use the concept of Fetcher now. I suppose it is because
Droids is designed to do more than web crawling and non-web
resource is not
"fetched"? In Droids 0.01, Protocol basically represent the
Fetcher (or at
least, Protocol+Worker)
- I strongly think we should use the term and concept of Fetcher
because it is a common terminology in crawler. Using common terms and
language makes our design more intuitive.
- Parser, Handler, Processor, Extractor etc. these are terms that
share very similar meaning. No matter how we use them, we need to give a
strict definition, e.g. in class level JavaDoc comment. My suggestions:
- Parser - the component that process the raw fetched Entity.
Output data is subject to implementation. One Entity will be
parsed by one
parser only.
- Extractor - the component to extract out link from entity.
Multiple extractors could be used for a parser. the primary
function is to
extract out link. user may also use it to do other extraction
or operation,
e.g. to store data in the Link, or just consume the parsed
data. A extract
depends directly to a parser. (we can't easily define a
contract between
Parser and Extractor, so let's do not attempt this.)
Extractor is a new concept. It is splitted from Parser and diff in
a way that each link shall be parsed once, and multiple extractor may
perform extraction or custom operation against the parsed
data. I think i
mentioned in another email before. Say when we use
NekoHtmlParser, we want
to parse just once, and maybe extractor1 is for extracting
outlink, and
extractor2 is for custom processing (such as indexing) and
both are based on
the same parsed data.
- Processor - too vague. do not use this concept, and we are not
using it anyway.
- Handler - for event based Parser, it may use a SAX
DefaultHandler. To avoid confusion, let's not to use handler in other
context.
- Entity
- I don't have any strong proposal and this section is just to
brainstorm some ideas. It would be good to clarify what we
want to achieve
in providing an Entity hierarchy, given that, the HC project actually
provides all the Entity already. Our entity is kind of a wrapper with
buffer.(and HC also have buffered entity) I guess we don't
want to depend on
HttpEntity from HC directly as Droids may touch entity beyond
HttpEntity,
e.g. File (but HC also as FileEntity...)
- Entity is the contract between Fetcher/Protocol and Parser. For
Entity, it's unlikely the user need to subclass it. if they
need to subclass
it, they also need to implement a Fetcher/Protocol + Parser.
The value of
sub-classing Entity is not significant. I suggest we just not
design it for
subclassing.
- Currently, we have a hierarchy of ManagedContentEntity,
ContentEntity, FileContentEntity, HttpContentEntity. For a
file parser and a
http parser, they can't easily use a common Entity interface
and I suppose
the parser implementation has to cast the entity. To me,
"ManagedContentEntity" doesn't give a lot of meaning than "Entity".
FileEntity and HttpEntity does make a different to me in
concept, but i
don't see how they could be related in implementation.
- My initial thought about the contract of parse is whether it can
just take a InputStream. And later i find it is necessary to
have a concept
of Entity that hold information like content/mime type,
encoding/charset,
size/length etc. But diff kind of entity just may have
different attribute
and it's not easy to define a comment contract. One of the
ideas in my mind
is to use a single final Entity class that extend HashMap.
- For HttpEntity, i do prefer to have a way to retrieve the
original HC HttpEntity object. (but it is unlikely we want to
expose that in
any interface) Notice that the wrapping make it a bit more complex in
constructing instance in unit test.
- Worker, Task, TaskMaster
- Make worker implements Runnable, Future.(and not RunnableFuture
for JDK5 compataibility) and we use run() as its main
interface. So it could
be use as a thread easier.
- I suggest to remove the concept of Task and TaskMaster. A
Droid/Crawler could do most work of the TaskMaster. These
concepts also
confuse with Thread, ThreadFactory, Executor, that creates
many similar
concepts.
- if we keep TaskMaster, i suggest to make it implements
ExecutorService, and we depends on Java util/concurrency API
rather than a
TaskMaster interface.
- Queue
- I suggest to remove the TaskQueue interface and use Queue<?
extends Link> as standard signature.
4. Link-centric design
- Link, extends HashMap, will act as a main arbitary data container, and
a vehicle that store attributes and data thoughout the whole lifecycle of
fetching, parsing, and extracting.
- if we do it extremely, all data can be stored as in the Link and all
interface could just use a single Link argument, e.g. parse(Link),
extract(Link). For sure it is not a good idea. So i make every
interface to
include the Link argument as well as key component. I found the extreme
usage is good in remote web service api, but not good in Java API.
- All components to be generic as <? extends Link>, user may use
another Link implementation for the whole Droid operation. an
example is a
WeightedLink.
- For a Link, only the URL is mandatory. A ID is needed for
implementing an in-memory set/hashtable to reject duplicated
Link quickly. I
suggest to make Link a class so people could create a link with new Link("
http://www.apache.org") easily, just like creating URL or URI.
5. Non-thread safe interface and fluent API
- take an example, insteaad of "Parse parse()", i suggest the parser to
store the parsed data inside itself, and we provide a reset() method to
clear the data for re-use. This design has pro and con.
- one of main pro is, we could simplify the model by omitting a Parse
class that is mainly for holding arbitrary data. And we also can't easily
define the return type of an interface. Take Fetcher as an example, a
fetcher typically contain a Request and Response object. Should we have
fetch() to return a FetchedData that has request, response, and
entity? it's
just a bit complex.
- I hope no one against Fluent API :-). with fluent api, it's like
"public Fetcher fetch()". And I don't always use Fluent Api,
only use when
it is good and the api call may be chained. e.g. parser.parse().getDate()
6. Factory and LinkMatcher design
- For worker, fetcher and parser, they are provided by users as a
Factory.
- For FetcherFactory and ParserFactory, new instance are created with
a newXXX(Link link)
- So, depends on the Link, the Factory will provide diff components.
e.g. for http link, it's a HttpFetcher, for File, it's a file
fetcher (not
impl.) for parer, it consults the content type.
- Every component implemnets a LinkMatcher interface that checks if a
Fecher/Parser/Extractor supports a particular link. This is primary for
automatic component registration without a need to explicitly providing a
mapping upfront. e.g. there might be a PNG parser that checks the
"contentType" attribute of the Link. The parser implemented the matches()
method. so we don't need to maintain a mapping hashmap between
contentType
and parser. The link matching may be complex so it's hard to use
a mapping
hashmap anyway. together with the filter framework, any
attribute could be
prepared by a filter first, so the factory could always rely on
the matcher
interface to find the correct parser/fetcher.
7. *Filter Framework
- This is a significant new concept. I propose to have a filter
framework that works as a chain for intercepting the works of every main
component. There are a main lifecycle filter that is named
Filter, and also
individual component filters such as FetchFilter and
ParseFilter. Lifecycle
filter is called by a Worker. Some works may not support it, e.g. my
WebServiceWorker that call GAE service do the whole batch of
fetch->parser->extract in one go, so there is no local filter.
Normal worker
shall call every filter after every operation. If the filter
return null, it
stop processing the link
- Filter
- When we have a confirmed lifecycle, e.g. poll a link from queue
-> fetch entity -> parse entity -> extract outlinks, then we
have a filter
that allow inteception in between every stage. e.g.
public Link polled(Link link)
public Fetcher fetched(Link link, Fetcher fetcher)
- any filter may influence the flow by changing the component
object like Fetcher/Parser, or they may return null and the
Worker/TaskMaster shall stop the process for that link.
- e.g. Duplicated Link handling could be done as a Filter. a
singleton NoRepeatFilter stores a Set of Link ID, and when any link is
extracted, it is check against the set and dupliated link
will be removed.
- It offers a lot of potentially such as providing runtime
statistics.
- Component filter
- e.g. FetchFilter, public void preFetch(Link, Fetcher),
postFetch(Link,Fetcher)
- component filter is expected to alter fetching behavior. e.g. for
preFetch for a http fetcher, the http request shall be
available already,
and the preFetch could as the fetcher to the concrete class,
and modify the
content of the HttpRequest before it is executed. e.g. to
append http header
/ cookie depends on any attribute in the Link.
- The global / lifecycle filter do filter after every component
operations. they are designed for different purpose.
8. Removed concepts based on the above proposal
- LinkTask, Task, just keep Link
- TaskMaster - with some refactor to assign responsibility to Droids
and Worker, the TaskMaster doesn't do too many things, and I suggest to
remove this concept.
- TaskQueue - just use Queue<Link>
- TaskQueueWithHistory - this is eliminated by the filter framework.
See the next section.
- TaskValidator - eliminate by the filter framework / implement as a
Filter
- URL Filter - could be implemented as a Filter
- Parse(Parse) - merged into Parser, I think "Parse" is a vague
concept and we would rather to have a Map return from than have a Parse
class
any comment?
regards,
mingfai
|