Fork me on GitHub

Apache Streams

Licensed under Apache License 2.0 - http://www.apache.org/licenses/LICENSE-2.0

    Release Notes - Streams - Version 0.5.1

Access

  • [STREAMS-563] - Remove "incubating" from all pom.xmls
  • [STREAMS-571] - Remove application.conf from release.sh

Bug

  • [STREAMS-494] - PropertyUtilTest is not running with surefire, and doesn't adequately test PropertyUtil
  • [STREAMS-495] - mvn -o clean can fail when streams-plugin-pojo snapshot is not installed
  • [STREAMS-499] - site:stage and site:publish problem with 0.5-incubating
  • [STREAMS-505] - Add ASFv2 Licenses
  • [STREAMS-506] - Missing ASFv2 License in Instagram Provider
  • [STREAMS-517] - compile error in streams-provider-instagram in master
  • [STREAMS-532] - ids_only=false for friends/followers does not work as expected in streams-provider-twitter
  • [STREAMS-545] - SchemaUtil.mergeProperties does not deep-copy / merge sub-objects
  • [STREAMS-549] - Bug in PropertyUtil.mergeProperties
  • [STREAMS-555] - Test in streams-util failing - because http://json-schema.org/schema has changed
  • [STREAMS-570] - Update enabled=false on TwitterThirtyDaySearchProviderIT

Improvement

  • [STREAMS-394] - Replace Travis CI with jenkins for building PRs
  • [STREAMS-492] - Create a twitter keyword search provider
  • [STREAMS-493] - streams-examples jenkins build fails
  • [STREAMS-502] - Re-implement instagram providers using juneau libraries
  • [STREAMS-508] - Utilize new third-party rest proxy capability in provider-twitter
  • [STREAMS-512] - Add @BeanProperty annotations in addition to @JsonProperty
  • [STREAMS-516] - switch @BeanProperty annotations to value
  • [STREAMS-519] - Support parsing dates with micro and nano precision with StreamsJacksonMapper
  • [STREAMS-521] - Add support for twitter direct message events and web hook events
  • [STREAMS-530] - Log the page_count, item_count, last_count when providers shut down
  • [STREAMS-531] - Implement the /friendships endpoint in streams-provider-twitter
  • [STREAMS-537] - Integration Tests should not assume presence of any specific system property values not controlled via SCM or maven lifecycle
  • [STREAMS-546] - Allow rat plugin to ignore .conf files in project root
  • [STREAMS-552] - Disable .debug() by default in org.apache.streams.twitter.api.Twitter
  • [STREAMS-557] - RiakHttpClient would be more useful if it exposed baseUrl publically
  • [STREAMS-565] - Permit access to StreamsConfigurator underlying Config without resolve()

New Feature

  • [STREAMS-535] - Implement /users/search and /users/suggestions in streams-provider-twitter
  • [STREAMS-553] - Implement Premium Search in streams-provider-twitter
  • [STREAMS-556] - Twitter: SevenDaySearchProvider
  • [STREAMS-566] - Twitter: ThirtyDaySearchProvider

Task

  • [STREAMS-316] - add “apache’ to the artifact name
  • [STREAMS-420] - reboot: cleanup JIRA
  • [STREAMS-496] - Remove twitter4j dependency from streams-provider-twitter
  • [STREAMS-497] - Add streams prefix to project and examples artifactIds
  • [STREAMS-500] - Remove jackson-datatype-json-org from project
  • [STREAMS-507] - Jenkins is not deploying snapshots for org/apache/streams/plugins
  • [STREAMS-522] - Remove mention of incubating status from code and web site
  • [STREAMS-523] - global dependency version bump
  • [STREAMS-524] - Ensure all Streams Components are serializable and implement the correct interface
  • [STREAMS-525] - discontinue use of reflections-maven
  • [STREAMS-526] - incorporate juneau 6.4.0 release
  • [STREAMS-527] - Move default values on configuration beans from json schemas into typesafe reference.conf files
  • [STREAMS-528] - disable style plugins by default
  • [STREAMS-529] - Make it possible to update data behind the StreamsConfigurator after initialization.
  • [STREAMS-533] - Fix JavaDoc links in streams-provider-twitter
  • [STREAMS-534] - Implement /search/tweets in streams-provider-twitter
  • [STREAMS-539] - quick profile should be disabled by default
  • [STREAMS-540] - drop dependence on third-party test data archive
  • [STREAMS-547] - add any missing AL headers
  • [STREAMS-548] - update juneau dependency to 7.0.0
  • [STREAMS-564] - Update dependency versions on juneau and httpclient 
    Release Notes - Streams - Version 0.5

Sub-task

  • [STREAMS-255] - Merge streams-master into streams-project

Bug

  • [STREAMS-257] - Remove incomplete DEPENDENCIES file from src zip artifact
  • [STREAMS-448] - TwitterUserInformationConfiguration only accepts Twitter IDs not usernames
  • [STREAMS-472] - Fix failing IT in streams-plugins
  • [STREAMS-477] - Fix link for checkstyle.xml
  • [STREAMS-481] - Update .travis.yml to use Java 8
  • [STREAMS-490] - Upgrade Flink to 1.2.0 and remove the remaining guava references in streams-examples

Improvement

  • [STREAMS-71] - Sysomos Provider should allow a target document or timeframe to be set
  • [STREAMS-341] - Cassandra Persistance
  • [STREAMS-344] - Support binary protocol in streams-persist-graph
  • [STREAMS-438] - Create coding guidelines document for website
  • [STREAMS-453] - tag each jira against one/more components.
  • [STREAMS-476] - Remove guava dependency from streams-example project

New Feature

  • [STREAMS-455] - LICENSE, NOTICE and DISCLAIMER in source control.
  • [STREAMS-478] - streams persist reader/writer for Apache Cassandra
  • [STREAMS-480] - Remove -incubating suffix from Release artifacts names
  • [STREAMS-482] - persist capability for riak
  • [STREAMS-483] - add support for SSL connections to streams-persist-cassandra
  • [STREAMS-484] - Update website to reflect deprecation of streams-master
  • [STREAMS-485] - Improve release process - no more repeated pasting of versions
  • [STREAMS-486] - Update release script on website

Task

  • [STREAMS-444] - tighten up test logging
  • [STREAMS-463] - Move every class in all repos underneath org.apache.streams

Test 

    Release Notes - Streams - Version 0.4

Sub-task

Bug

  • [STREAMS-73] - Interfaces
  • [STREAMS-132] - RegexUtils does not ensure that content is non-null
  • [STREAMS-151] - Refactor Facebook Provider to have continuous AND finite mode
  • [STREAMS-159] - Add facebook page feed provider
  • [STREAMS-177] - new NPE in LinkResolver causing surefire to fail
  • [STREAMS-258] - Prevent signing of .asc, .m5, and .sha1 artifacts
  • [STREAMS-391] - streams-provider-dropwizard exceptions in test logs
  • [STREAMS-405] - Link to src/site/markdown/index.md in README.md is broken
  • [STREAMS-407] - add DateTimeSerializers for formats safely, don't crash if invalid
  • [STREAMS-414] - Incorrect Documentation
  • [STREAMS-436] - Put a timeout on all Provider Integration Tests
  • [STREAMS-437] - DatumFromMetadataProcessorIT failing during 0.4 release
  • [STREAMS-446] - RAT check fails in prep for 0.4-incubating release
  • [STREAMS-447] - Scala-plugin failures in prep for 0.4-incubating release

Improvement

  • [STREAMS-127] - JsonSchema Replication in Datasift provider
  • [STREAMS-160] - Embed original source provider pojos inside datasift pojos if possible
  • [STREAMS-186] - Platform-level 'detectConfiguration'
  • [STREAMS-399] - Add any missing fields to tweet.json
  • [STREAMS-400] - streams-persist-elasticsearch : bump to version 2.x
  • [STREAMS-403] - Ensure all providers function stand-alone
  • [STREAMS-404] - Integration Test for FsElasticsearchIndex
  • [STREAMS-411] - ability (and instructions on how) to run providers directly from console
  • [STREAMS-413] - Update dependency and plugin versions - Q4 2016
  • [STREAMS-425] - better tracking completion in multi-threaded providers
  • [STREAMS-426] - streams-persist-mongo : test with docker
  • [STREAMS-427] - Support any jackson-compatible class as valid input to base converters and provider converters
  • [STREAMS-428] - Update example markdown on website
  • [STREAMS-429] - fix failing tests in streams-plugins
  • [STREAMS-430] - update jenkins to run advanced integration testing
  • [STREAMS-431] - Remove streams.util.RegexUtils
  • [STREAMS-432] - Update to Java 8
  • [STREAMS-433] - Upgrade maven and jenkins to build with jdk8
  • [STREAMS-434] - Delete CustomDateTimeFormat which is not used anywhere
  • [STREAMS-435] - remove incubator-streams-master-pom.xml

New Feature

  • [STREAMS-213] - Publish jsonschemas to a web-accessible URL when jenkins builds snapshot and releases
  • [STREAMS-389] - Support generation of scala source from jsonschemas
  • [STREAMS-398] - Support generation of hive table definitions from jsonschema
  • [STREAMS-418] - Flink twitter example(s)

Story

Task

  • [STREAMS-203] - Update GooglePlus TypeConverter to handle Post Activities
  • [STREAMS-316] - add “apache’ to the artifact name
  • [STREAMS-408] - Check package names and run instructions of modules in streams-examples/local
  • [STREAMS-409] - the copyright year in the NOTICE files need to be updated for 2016.
  • [STREAMS-410] - Delete any modules which have been removed from reactor from master branch
  • [STREAMS-416] - Delete defunct or not-implemented provider modules
  • [STREAMS-417] - Collect example AS 2.0 object and activity documents to use in test cases
  • [STREAMS-419] - reboot: cleanup git branches and tags
  • [STREAMS-421] - Delete defunct or not-implemented runtime modules

Test

  • [STREAMS-415] - Proof of concept integration test that pulls actual data from generator 
    Release Notes - Streams - Version 0.3

Sub-task

Bug

  • [STREAMS-158] - Sysomos Processor exceptions result in failed processor thread
  • [STREAMS-220] - FacebookPostSerializer is broken
  • [STREAMS-223] - streams-monitoring exception when streamConfig not set
  • [STREAMS-227] - Array out of bounds Exception running FacebookTypeConverter
  • [STREAMS-229] - GooglePlus TypeConverter needs to be able to accommodate String datums
  • [STREAMS-230] - Broadcast Monitor doesn't start in all cases
  • [STREAMS-236] - AbstractRegexExtensionExtractor should not allow duplicate entities
  • [STREAMS-260] - FacebookPageFeedDataCollector should handle backoff strategy correctly
  • [STREAMS-264] - LinkExpansion Tests are depending on (non-existent) external resources
  • [STREAMS-311] - TwitterUserInformationProvider stalls with > 20 items provided
  • [STREAMS-337] - WebHdfsPersistReader/Writer should allow user-specified file encoding
  • [STREAMS-338] - Runtime exceptions caught by LocalStreamBuilder aren't logged
  • [STREAMS-346] - persist-hdfs: unnecessary port config causes failure when reading from local filesystem
  • [STREAMS-347] - persist-hdfs: incorrectly escapes output when write called on a json String
  • [STREAMS-354] - pojo.json.Collection.totalItems should be an integer
  • [STREAMS-355] - WebHdfsPersistReader should be serializable
  • [STREAMS-356] - Metadata missing in StreamsDatum constructors and toString
  • [STREAMS-357] - Streams can shutdown before starting if providers haven't started yet when runtime first checks run status
  • [STREAMS-358] - Refreshing indexes in EsPW.cleanUp can result in streams not terminating
  • [STREAMS-374] - Fix ActivityConverterUtil detectClasses(document) exception handling
  • [STREAMS-377] - can't create different LineReadWriteUtils
  • [STREAMS-380] - Media_Link should be serializable
  • [STREAMS-382] - Media_Link width and height should be integers, not floats
  • [STREAMS-386] - Clean up streams-pojo jsonschemas to ensure all beans are serialiazable
  • [STREAMS-395] - incubator-streams-examples COMPILATION ERROR

Improvement

  • [STREAMS-126] - StreamsLocalBuilder should look for provider timeout in typesafe config
  • [STREAMS-191] - Streams implement use of throughput queues
  • [STREAMS-226] - Consolidate all stream-wide configuration
  • [STREAMS-279] - Create Youtube User/Channel provider
  • [STREAMS-294] - Allow for custom setting of QueueSize, BatchSize, and ScrollTimeout
  • [STREAMS-308] - Update readme and website to recommend latest version of JDK
  • [STREAMS-313] - Performant Off-line Neo4j GraphPersistWriter
  • [STREAMS-320] - Update release documentation to cover build and deploy of mvn site to svn
  • [STREAMS-321] - Add HOCON Converter support for input object not at root node
  • [STREAMS-322] - Support gzipped files in WebHdfsReader
  • [STREAMS-324] - clean up example poms
  • [STREAMS-325] - Configure monitoring via typesafe
  • [STREAMS-326] - Bump hadoop version in streams-persist-hdfs
  • [STREAMS-328] - StreamsJacksonMapper should omit null/empty fields when serializing
  • [STREAMS-329] - Processor to derive required ES metadata from standard fields
  • [STREAMS-330] - make TwitterErrorHandler respect the Twitter4j RateLimitStatus reset time
  • [STREAMS-332] - Restore ability to test data conversions on flat files during build
  • [STREAMS-333] - ElasticsearchPersistUpdater should set parent+routing when available
  • [STREAMS-335] - Publish schema and configuration resources to maven site pages
  • [STREAMS-336] - consolidate common pom sections to streams-master
  • [STREAMS-339] - incorporate defined extension fields for activities and actors into pojos
  • [STREAMS-342] - Expose convertResultToString and processLine as public methods
  • [STREAMS-343] - Make SerializationUtil spark-compatible
  • [STREAMS-348] - Add get/set methods for StreamsConfiguration to Builder interface
  • [STREAMS-350] - provider-twitter: derive baseURL from configuration
  • [STREAMS-353] - Support removal of 'extensions' from document path structure
  • [STREAMS-361] - Persist Reader/Writer for Amazon Kinesis
  • [STREAMS-362] - Make shutdown timers in local runtime fully configurable
  • [STREAMS-363] - upgrade persist-s3 to match persist-hdfs features
  • [STREAMS-364] - Allow resolution of typesafe/stream config from url in stream-config
  • [STREAMS-368] - Add String getId() to StreamsOperation
  • [STREAMS-370] - Support arbitrary labels in streams-persist-graph
  • [STREAMS-371] - Allow use of ids endpoints in TwitterFollowingProvider
  • [STREAMS-373] - Allow specification of 'maximum_items' in twitter providers
  • [STREAMS-375] - Override component configuration when valid beans supplied by runtime as prepare args
  • [STREAMS-376] - When twitter providers fail authentication, log the ID that could not be accessed.
  • [STREAMS-378] - incompatibility between streams binaries and spark 1.5
  • [STREAMS-381] - streams-provider-twitter: User can contain a Status
  • [STREAMS-383] - register DefaultScalaModule in StreamsJacksonMapper
  • [STREAMS-387] - Default behavior of streams-pojo-extensions: use additionalProperties directly
  • [STREAMS-390] - Eliminate dependency and plugin warnings from build
  • [STREAMS-392] - Centralized logging configuration for maven build
  • [STREAMS-393] - Switch any usage of System.out and System.err to slf4j
  • [STREAMS-397] - streams-provider-youtube test failure in jenkins
  • [STREAMS-401] - Refresh Streams Website
  • [STREAMS-406] - Bring streams-master markdown files into compliance with rat plugin

New Feature

Story

  • [STREAMS-43] - Complete, test, and document g+ provider
  • [STREAMS-44] - Complete, test, and document sysomos provider
  • [STREAMS-46] - Complete, test, and document facebook API provider
  • [STREAMS-261] - Create Facebook Bio Collector/Provider
  • [STREAMS-272] - Create a Youtube Post Provider

Task

  • [STREAMS-123] - Add twitter specific link handling for datasift
  • [STREAMS-202] - Create Google Plus Deserializer and TypeConverter
  • [STREAMS-274] - Create a YoutubeTypeConverter and serializer
  • [STREAMS-280] - Add ability to get a final document count from the Sysomos Provider
  • [STREAMS-298] - Ensure jcl-over-slf4j bridge is included where necessary
  • [STREAMS-317] - release a .tar.gz artefact as well as a .zip
  • [STREAMS-318] - remove test files whose license may not be compatible with an Apache release (google-gplus)
  • [STREAMS-319] - update <developer> info in the pom with active contributors
  • [STREAMS-331] - Set up build of master and pull requested branches on travis-ci.org
  • [STREAMS-334] - version bump datasift module to remove boundary repo dependency
  • [STREAMS-351] - add A2 license to site.xml files
  • [STREAMS-359] - copy streams-master pom into streams-project as CI workaround
  • [STREAMS-372] - Support deploy to snapshot repo by CI

Test

  • [STREAMS-208] - Integration Testing capability and reference Integration Test
  • [STREAMS-384] - TestLinkUnwinderProcessor.test404Link is failing 
    Release Notes - Streams - Version 0.2

Sub-task

  • [STREAMS-275] - ActivityConverterProcessor should apply reflection mode when configuration is not provided
  • [STREAMS-277] - Upgrade streams-provider-twitter to work with reflection-based conversion
  • [STREAMS-305] - Add missing AL
  • [STREAMS-306] - Intermittent test failures
  • [STREAMS-307] - Release test-jar packaging
  • [STREAMS-309] - incubator-streams-examples site plugin
  • [STREAMS-312] - Remove test resources without clear licensing and ignore tests that require them

Bug

  • [STREAMS-155] - Build hadoop modules with Apache artifacts
  • [STREAMS-200] - StreamsProcessorTask ignores any processing
  • [STREAMS-225] - Streams need to remove any of their JMX beans on shutdown/cleanup
  • [STREAMS-243] - S3 Persist Writer does not flush or shutdown on stream shutdown
  • [STREAMS-263] - FacebookTypeConverter should be able to handle Facebook Pages
  • [STREAMS-266] - some classes/tests are using the wrong NotImplementedException
  • [STREAMS-278] - Rework pig runtime as part of switch from 'Serializer' to 'Converter'
  • [STREAMS-281] - enable BroadcastMessagePersister
  • [STREAMS-288] - StreamsJacksonModule should not scan for DateTimeFormats by default
  • [STREAMS-296] - Local Runtime doesn't allow persist writers enough time to flush and close during shutdown
  • [STREAMS-299] - Sysomos Provider uses dev API URL

Improvement

  • [STREAMS-147] - Platform-level type conversion
  • [STREAMS-201] - Util function to remove all MXBeans for tests
  • [STREAMS-214] - Create, test, and document file-backed persistance module
  • [STREAMS-271] - suggest increasing JVM heap in readme
  • [STREAMS-273] - Support POST endpoints in streams-http
  • [STREAMS-284] - Read/write parent IDs in streams-persist-elasticsearch
  • [STREAMS-285] - Add all objectTypes in spec to streams-pojo
  • [STREAMS-286] - Add all verbs in spec to streams-pojo
  • [STREAMS-293] - allow for missing metadata fields in streams-persist-hdfs

New Feature

Story

  • [STREAMS-47] - Complete, test, and document mongo persist

Task

  • [STREAMS-95] - TwitterProfileProcessor needs to send a user ID in each StreamsDatum
  • [STREAMS-304] - Perform 0.2-incubating release 
    Release Notes - Streams - Version 0.1

Sub-task

  • [STREAMS-212] - Generic Type Converter Processor
  • [STREAMS-215] - Add method to StreamsConfigurator to return Serializable config given a Typesafe Config
  • [STREAMS-218] - Generic Activity Converter Processor
  • [STREAMS-241] - Reflection-based StreamsJacksonMapper
  • [STREAMS-245] - Clean-up root POM
  • [STREAMS-246] - Clean-up module POMs
  • [STREAMS-247] - Clean-up READMEs
  • [STREAMS-248] - Bring website content in-line with released capabilities.
  • [STREAMS-249] - Javadoc plugin
  • [STREAMS-250] - Prepare release notes
  • [STREAMS-251] - Dry-run release process, review artifacts, update documentation
  • [STREAMS-254] - Resolve rat plugin failures
  • [STREAMS-255] - Merge streams-master into streams-project
  • [STREAMS-256] - streams-components-all version doesn't bump during release
  • [STREAMS-267] - .gitignore eclipse workspace files
  • [STREAMS-268] - remove streams-web from master
  • [STREAMS-269] - maven-remote-resources-plugin using SNAPSHOT resource-bundles

Bug

  • [STREAMS-167] - TwitterConfigurator doesn't properly create TwitterUserInfoConfiguration from hocon
  • [STREAMS-219] - src/main/resource files are being created by build
  • [STREAMS-252] - Monitor Executor Service in LocalStreamBuilder needs to be Flexible
  • [STREAMS-262] - jars in runner folder of source release
  • [STREAMS-265] - some jar artifacts from 0.1-rc1 did not contain DISCLAIMER

Improvement

  • [STREAMS-68] - ActivitySerializer should be ActivityConverter
  • [STREAMS-143] - Allow modules to get instances of StreamsJacksonMapper that can process additional DateTime formats
  • [STREAMS-244] - Prepare for 0.1 release