Hadoop 2.7.0 Release Notes
These release notes include new developer and user-facing incompatibilities, features, and major improvements.
Changes since Hadoop 2.6.0
- YARN-3430.
Blocker sub-task reported by Xuan Gong and fixed by Xuan Gong (resourcemanager , webapp , yarn)
RMAppAttempt headroom data is missing in RM Web UI
- YARN-3393.
Critical bug reported by Zhijie Shen and fixed by Zhijie Shen (timelineserver)
Getting application(s) goes wrong when app finishes before starting the attempt
- YARN-3384.
Minor bug reported by Naganarasimha G R and fixed by Naganarasimha G R
TestLogAggregationService.verifyContainerLogs fails after YARN-2777
- YARN-3379.
Major sub-task reported by Xuan Gong and fixed by Xuan Gong (resourcemanager , webapp , yarn)
Missing data in localityTable and ResourceRequests table in RM WebUI
- YARN-3369.
Blocker bug reported by Giovanni Matteo Fumarola and fixed by Brahma Reddy Battula (resourcemanager)
Missing NullPointer check in AppSchedulingInfo causes RM to die
- YARN-3349.
Minor improvement reported by zhihai xu and fixed by zhihai xu (test)
Treat all exceptions as failure in TestFSRMStateStore#testFSRMStateStoreClientRetry
- YARN-3338.
Blocker bug reported by Zhijie Shen and fixed by Zhijie Shen (build)
Exclude jline dependency from YARN
- YARN-3336.
Critical bug reported by zhihai xu and fixed by zhihai xu (resourcemanager)
FileSystem memory leak in DelegationTokenRenewer
- YARN-3304.
Blocker bug reported by Junping Du and fixed by Junping Du (nodemanager)
ResourceCalculatorProcessTree#getCpuUsagePercent default return value is inconsistent with other getters
- YARN-3300.
Major sub-task reported by Xuan Gong and fixed by Xuan Gong (resourcemanager)
outstanding_resource_requests table should not be shown in AHS
- YARN-3296.
Major bug reported by Hitesh Shah and fixed by Hitesh Shah
yarn.nodemanager.container-monitor.process-tree.class is configurable but ResourceCalculatorProcessTree class is marked Private
- YARN-3295.
Trivial bug reported by Masatake Iwasaki and fixed by Masatake Iwasaki (documentation)
Fix documentation nits found in markdown conversion
- YARN-3287.
Major bug reported by Jonathan Eagles and fixed by Daryn Sharp
TimelineClient kerberos authentication failure uses wrong login context.
- YARN-3285.
Major improvement reported by Masatake Iwasaki and fixed by Masatake Iwasaki (documentation)
Convert branch-2 .apt.vm files of YARN to markdown
- YARN-3281.
Minor bug reported by Chengbing Liu and fixed by Chengbing Liu (scripts)
Add RMStateStore to StateMachine visualization list
- YARN-3275.
Major bug reported by Eric Payne and fixed by Eric Payne
CapacityScheduler: Preemption happening on non-preemptable queues
- YARN-3273.
Major improvement reported by Jian He and fixed by Rohith
Improve web UI to facilitate scheduling analysis and debugging
- YARN-3272.
Major improvement reported by Jian He and fixed by Jian He
Surface container locality info in RM web UI
- YARN-3270.
Minor bug reported by Rohit Agarwal and fixed by Rohit Agarwal
node label expression not getting set in ApplicationSubmissionContext
- YARN-3267.
Major bug reported by Prakash Ramachandran and fixed by Chang Li
Timelineserver applies the ACL rules after applying the limit on the number of records
- YARN-3265.
Blocker sub-task reported by Wangda Tan and fixed by Wangda Tan (capacityscheduler , resourcemanager)
CapacityScheduler deadlock when computing absolute max avail capacity (fix for trunk/branch-2)
- YARN-3262.
Major improvement reported by Jian He and fixed by Jian He (yarn)
Surface application outstanding resource requests table in RM web UI
- YARN-3256.
Major bug reported by Anubhav Dhoot and fixed by Anubhav Dhoot
TestClientToAMTokens#testClientTokenRace is not running against all Schedulers even when using ParameterizedSchedulerTestBase
- YARN-3255.
Major bug reported by Konstantin Shvachko and fixed by Konstantin Shvachko (nodemanager , resourcemanager)
RM, NM, JobHistoryServer, and WebAppProxyServer's main() should support generic options
- YARN-3249.
Minor improvement reported by Ryu Kobayashi and fixed by Ryu Kobayashi (resourcemanager)
Add a "kill application" button to Resource Manager's Web UI
- YARN-3242.
Critical bug reported by zhihai xu and fixed by zhihai xu (resourcemanager)
Asynchrony in ZK-close can lead to ZKRMStateStore watcher receiving events for old client
- YARN-3239.
Major bug reported by Hitesh Shah and fixed by Jian He
WebAppProxy does not support a final tracking url which has query fragments and params
- YARN-3238.
Blocker bug reported by Jason Lowe and fixed by Jason Lowe
Connection timeouts to nodemanagers are retried at multiple levels
- YARN-3237.
Major bug reported by Rushabh S Shah and fixed by Rushabh S Shah
AppLogAggregatorImpl fails to log error cause
- YARN-3236.
Trivial improvement reported by zhihai xu and fixed by zhihai xu (resourcemanager)
cleanup RMAuthenticationFilter#AUTH_HANDLER_PROPERTY.
- YARN-3231.
Critical bug reported by Siqi Li and fixed by Siqi Li
FairScheduler: Changing queueMaxRunningApps interferes with pending jobs
- YARN-3230.
Major improvement reported by Jian He and fixed by Jian He
Clarify application states on the web UI
- YARN-3227.
Critical bug reported by Jonathan Eagles and fixed by Zhijie Shen
Timeline renew delegation token fails when RM user's TGT is expired
- YARN-3222.
Critical bug reported by Rohith and fixed by Rohith (resourcemanager)
RMNodeImpl#ReconnectNodeTransition should send scheduler events in sequential order
- YARN-3217.
Major task reported by Akira AJISAKA and fixed by Brahma Reddy Battula
Remove httpclient dependency from hadoop-yarn-server-web-proxy
- YARN-3207.
Major bug reported by Prakash Ramachandran and fixed by Zhijie Shen (timelineserver)
secondary filter matches entites which do not have the key being filtered for.
- YARN-3203.
Minor improvement reported by Brahma Reddy Battula and fixed by Brahma Reddy Battula
Correct a log message in AuxServices
- YARN-3195.
Minor improvement reported by Jagadesh Kiran N and fixed by Jagadesh Kiran N (client)
Add -help to yarn logs and nodes CLI command
- YARN-3194.
Blocker bug reported by Rohith and fixed by Rohith (resourcemanager)
RM should handle NMContainerStatuses sent by NM while registering if NM is Reconnected node
- YARN-3191.
Trivial bug reported by Rohith and fixed by Rohith (nodemanager)
Log object should be initialized with its own class
- YARN-3187.
Major improvement reported by Naganarasimha G R and fixed by Gururaj Shetty (capacityscheduler , documentation)
Documentation of Capacity Scheduler Queue mapping based on user or group
- YARN-3182.
Minor improvement reported by Ray Chiang and fixed by Ray Chiang
Cleanup switch statement in ApplicationMasterLauncher#handle()
- YARN-3179.
Minor improvement reported by Ray Chiang and fixed by Ray Chiang
Update use of Iterator to Iterable
- YARN-3171.
Minor sub-task reported by Jeff Zhang and fixed by Naganarasimha G R (timelineserver)
Sort by Application id, AppAttempt & ContainerID doesn't work in ATS / RM web ui
- YARN-3164.
Minor bug reported by Bibin A Chundatt and fixed by Bibin A Chundatt (resourcemanager)
rmadmin command usage prints incorrect command name
- YARN-3160.
Major bug reported by Chengbing Liu and fixed by Chengbing Liu (resourcemanager)
Non-atomic operation on nodeUpdateQueue in RMNodeImpl
- YARN-3158.
Major improvement reported by Devaraj K and fixed by Varun Saxena
Correct log messages in ResourceTrackerService
- YARN-3157.
Minor improvement reported by Bibin A Chundatt and fixed by Bibin A Chundatt (resourcemanager)
Refactor the exception handling in ConverterUtils#to*Id
- YARN-3155.
Minor bug reported by Li Lu and fixed by Li Lu
Refactor the exception handling code for TimelineClientImpl's retryOn method
- YARN-3154.
Blocker sub-task reported by Xuan Gong and fixed by Xuan Gong (nodemanager , resourcemanager)
Should not upload partial logs for MR jobs or other "short-running' applications
- YARN-3151.
Minor bug reported by Bibin A Chundatt and fixed by Rohith (client , resourcemanager)
On Failover tracking url wrong in application cli for KILLED application
- YARN-3149.
Trivial bug reported by Bibin A Chundatt and fixed by Bibin A Chundatt (resourcemanager)
Typo in message for invalid application id
- YARN-3147.
Major improvement reported by Steve Loughran and fixed by Steve Loughran (webapp)
Clean up RM web proxy code
- YARN-3145.
Major bug reported by Jian He and fixed by Tsuyoshi Ozawa
ConcurrentModificationException on CapacityScheduler ParentQueue#getQueueUserAclInfo
- YARN-3144.
Major improvement reported by Jonathan Eagles and fixed by Jonathan Eagles
Configuration for making delegation token failures to timeline server not-fatal
- YARN-3143.
Major bug reported by Kendall Thrapp and fixed by Jason Lowe (webapp)
RM Apps REST API can return NPE or entries missing id and other fields
- YARN-3132.
Major sub-task reported by Wangda Tan and fixed by Wangda Tan (api , client , resourcemanager)
RMNodeLabelsManager should remove node from node-to-label mapping when node becomes deactivated
- YARN-3131.
Major bug reported by Chang Li and fixed by Chang Li
YarnClientImpl should check FAILED and KILLED state in submitApplication
- YARN-3124.
Major sub-task reported by Wangda Tan and fixed by Wangda Tan (api , client , resourcemanager)
Capacity Scheduler LeafQueue/ParentQueue should use QueueCapacities to track capacities-by-label
- YARN-3123.
Major improvement reported by Zhijie Shen and fixed by Naganarasimha G R (client)
Make YARN CLI show a single completed container even if the app is running
- YARN-3122.
Major sub-task reported by Anubhav Dhoot and fixed by Anubhav Dhoot (nodemanager)
Metrics for container's actual CPU usage
- YARN-3113.
Major bug reported by Chang Li and fixed by Steve Loughran
Release audit warning for "Sorting icons.psd"
- YARN-3108.
Major improvement reported by Chang Li and fixed by Chang Li
ApplicationHistoryServer doesn't process -D arguments
- YARN-3104.
Major bug reported by Jason Lowe and fixed by Jason Lowe (resourcemanager)
RM generates new AMRM tokens every heartbeat between rolling and activation
- YARN-3103.
Blocker bug reported by Jason Lowe and fixed by Jason Lowe (client)
AMRMClientImpl does not update AMRM token properly
- YARN-3101.
Major bug reported by Anubhav Dhoot and fixed by Anubhav Dhoot (fairscheduler)
In Fair Scheduler, fix canceling of reservations for exceeding max share
- YARN-3100.
Major improvement reported by Jian He and fixed by Jian He
Make YARN authorization pluggable
- YARN-3099.
Major sub-task reported by Wangda Tan and fixed by Wangda Tan (api , client , resourcemanager)
Capacity Scheduler LeafQueue/ParentQueue should use ResourceUsage to track used-resources-by-label.
- YARN-3098.
Major sub-task reported by Wangda Tan and fixed by Wangda Tan (capacityscheduler)
Create common QueueCapacities class in Capacity Scheduler to track capacities-by-labels of queues
- YARN-3094.
Major bug reported by Jun Gong and fixed by Jun Gong (resourcemanager)
reset timer for liveness monitors after RM recovery
- YARN-3092.
Major sub-task reported by Wangda Tan and fixed by Wangda Tan (api , client , resourcemanager)
Create common ResourceUsage class to track labeled resource usages in Capacity Scheduler
- YARN-3090.
Major bug reported by Jason Lowe and fixed by Varun Saxena (nodemanager)
DeletionService can silently ignore deletion task failures
- YARN-3089.
Blocker bug reported by Jason Lowe and fixed by Eric Payne
LinuxContainerExecutor does not handle file arguments to deleteAsUser
- YARN-3088.
Major bug reported by Jason Lowe and fixed by Eric Payne (nodemanager)
LinuxContainerExecutor.deleteAsUser can throw NPE if native executor returns an error
- YARN-3086.
Minor improvement reported by Robert Metzger and fixed by Robert Metzger (test)
Make NodeManager memory configurable in MiniYARNCluster
- YARN-3085.
Major improvement reported by Jason Lowe and fixed by Rohith (resourcemanager)
Application summary should include the application type
- YARN-3082.
Major bug reported by Anubhav Dhoot and fixed by Anubhav Dhoot
Non thread safe access to systemCredentials in NodeHeartbeatResponse processing
- YARN-3079.
Major bug reported by zhihai xu and fixed by zhihai xu
Scheduler should also update maximumAllocation when updateNodeResource.
- YARN-3078.
Minor bug reported by sam liu and fixed by (log-aggregation)
LogCLIHelpers lacks of a blank space before string 'does not exist'
- YARN-3077.
Major improvement reported by Chun Chen and fixed by Chun Chen (resourcemanager)
RM should create yarn.resourcemanager.zk-state-store.parent-path recursively
- YARN-3076.
Major sub-task reported by Varun Saxena and fixed by Varun Saxena (client)
Add API/Implementation to YarnClient to retrieve label-to-node mapping
- YARN-3075.
Major sub-task reported by Varun Saxena and fixed by Varun Saxena (resourcemanager)
NodeLabelsManager implementation to retrieve label to node mapping
- YARN-3074.
Major bug reported by Jason Lowe and fixed by Varun Saxena (nodemanager)
Nodemanager dies when localizer runner tries to write to a full disk
- YARN-3071.
Trivial bug reported by Masatake Iwasaki and fixed by Masatake Iwasaki (documentation)
Remove invalid char from sample conf in doc of FairScheduler
- YARN-3070.
Minor test reported by Ted Yu and fixed by Junping Du
TestRMAdminCLI#testHelp fails for transitionToActive command
- YARN-3064.
Critical bug reported by Wangda Tan and fixed by Jian He (scheduler)
TestRMRestart/TestContainerResourceUsage/TestNodeManagerResync failure with allocation timeout
- YARN-3058.
Minor bug reported by Yi Liu and fixed by Yi Liu
Fix error message of tokens' activation delay configuration
- YARN-3056.
Trivial improvement reported by zhihai xu and fixed by zhihai xu (test)
add verification for containerLaunchDuration in TestNodeManagerMetrics.
- YARN-3055.
Blocker bug reported by Yi Liu and fixed by Daryn Sharp (security)
The token is not renewed properly if it's shared by jobs (oozie) in DelegationTokenRenewer
- YARN-3029.
Major bug reported by Steve Loughran and fixed by Varun Saxena (nodemanager)
FSDownload.unpack() uses local locale for FS case conversion, may not work everywhere
- YARN-3028.
Major sub-task reported by Jian He and fixed by Rohith (api , client , resourcemanager)
Better syntax for replaceLabelsOnNode in RMAdmin CLI
- YARN-3027.
Major bug reported by Anubhav Dhoot and fixed by Anubhav Dhoot
Scheduler should use totalAvailable resource from node instead of availableResource for maxAllocation
- YARN-3024.
Major bug reported by Chengbing Liu and fixed by Chengbing Liu (nodemanager)
LocalizerRunner should give DIE action when all resources are localized
- YARN-3022.
Major improvement reported by Anubhav Dhoot and fixed by Anubhav Dhoot
Expose Container resource information from NodeManager for monitoring
- YARN-3019.
Major sub-task reported by Jian He and fixed by Jian He (resourcemanager)
Make work-preserving-recovery the default mechanism for RM recovery
- YARN-3015.
Minor bug reported by Chris Nauroth and fixed by Varun Saxena (scripts)
yarn classpath command should support same options as hadoop classpath.
- YARN-3014.
Major sub-task reported by Wangda Tan and fixed by Wangda Tan
Replaces labels on a host should update all NM's labels on that host
- YARN-3011.
Major sub-task reported by Wang Hao and fixed by Varun Saxena (nodemanager)
NM dies because of the failure of resource localization
- YARN-3010.
Minor bug reported by Yi Liu and fixed by Yi Liu
Fix recent findbug issue in AbstractYarnScheduler
- YARN-3005.
Trivial improvement reported by Akira AJISAKA and fixed by
[JDK7] Use switch statement for String instead of if-else statement in RegistrySecurity.java
- YARN-2998.
Major sub-task reported by Anubhav Dhoot and fixed by Anubhav Dhoot (fairscheduler)
Abstract out scheduler independent PlanFollower components
- YARN-2997.
Major bug reported by Chengbing Liu and fixed by Chengbing Liu (nodemanager)
NM keeps sending already-sent completed containers to RM until containers are removed from context
- YARN-2996.
Major improvement reported by Yi Liu and fixed by Yi Liu (resourcemanager)
Refine fs operations in FileSystemRMStateStore and few fixes
- YARN-2994.
Major sub-task reported by Jian He and fixed by Jian He (resourcemanager)
Document work-preserving RM restart
- YARN-2993.
Major bug reported by Yi Liu and fixed by Yi Liu (resourcemanager)
Several fixes (missing acl check, error log msg ...) and some refinement in AdminService
- YARN-2992.
Blocker bug reported by Karthik Kambatla and fixed by Karthik Kambatla (resourcemanager)
ZKRMStateStore crashes due to session expiry
- YARN-2991.
Blocker bug reported by Zhijie Shen and fixed by Rohith
TestRMRestart.testDecomissionedNMsMetricsOnRMRestart intermittently fails on trunk
- YARN-2990.
Major bug reported by Karthik Kambatla and fixed by Karthik Kambatla (fairscheduler)
FairScheduler's delay-scheduling always waits for node-local and rack-local delays, even for off-rack-only requests
- YARN-2988.
Minor bug reported by Ted Yu and fixed by Ted Yu
Graph#save() may leak file descriptors
- YARN-2987.
Major bug reported by Jian He and fixed by Varun Saxena
ClientRMService#getQueueInfo doesn't check app ACLs
- YARN-2984.
Major sub-task reported by Karthik Kambatla and fixed by Karthik Kambatla (nodemanager)
Metrics for container's actual memory usage
- YARN-2983.
Major bug reported by Varun Saxena and fixed by Varun Saxena (resourcemanager)
NPE possible in ClientRMService#getQueueInfo
- YARN-2978.
Critical bug reported by Jason Tufo and fixed by Varun Saxena
ResourceManager crashes with NPE while getting queue info
- YARN-2977.
Major bug reported by Junping Du and fixed by Junping Du
TestNMClient get failed intermittently
- YARN-2975.
Blocker bug reported by Karthik Kambatla and fixed by Karthik Kambatla
FSLeafQueue app lists are accessed without required locks
- YARN-2972.
Major bug reported by Jason Lowe and fixed by Jason Lowe (resourcemanager)
DelegationTokenRenewer thread pool never expands
- YARN-2971.
Major sub-task reported by Jonathan Eagles and fixed by Jonathan Eagles (timelineserver)
RM uses conf instead of token service address to renew timeline delegation tokens
- YARN-2970.
Minor sub-task reported by Junping Du and fixed by Varun Saxena (api , client , resourcemanager)
NodeLabel operations in RMAdmin CLI get missing in help command.
- YARN-2964.
Blocker bug reported by Daryn Sharp and fixed by Jian He (resourcemanager)
RM prematurely cancels tokens for jobs that submit jobs (oozie)
- YARN-2958.
Blocker bug reported by Zhijie Shen and fixed by Varun Saxena (resourcemanager)
RMStateStore seems to unnecessarily and wrongly store sequence number separately
- YARN-2957.
Minor improvement reported by Ray Chiang and fixed by Ray Chiang
Create unit test to automatically compare YarnConfiguration and yarn-default.xml
- YARN-2956.
Minor bug reported by Remus Rusanu and fixed by Masatake Iwasaki (documentation)
Some yarn-site index linked pages are difficult to discover because are not in the side bar
- YARN-2952.
Major bug reported by Jian He and fixed by Rohith
Incorrect version check in RMStateStore
- YARN-2950.
Minor improvement reported by Harsh J and fixed by Dustin Cote (webapp)
Change message to mandate, not suggest JS requirement on UI
- YARN-2949.
Major task reported by Varun Vasudev and fixed by Varun Vasudev (documentation , nodemanager)
Add documentation for CGroups
- YARN-2945.
Major bug reported by Tsuyoshi Ozawa and fixed by Tsuyoshi Ozawa
FSLeafQueue#assignContainer - document the reason for using both write and read locks
- YARN-2944.
Minor sub-task reported by Chris Trezzo and fixed by Chris Trezzo
InMemorySCMStore can not be instantiated with ReflectionUtils#newInstance
- YARN-2943.
Major sub-task reported by Wangda Tan and fixed by Wangda Tan (resourcemanager)
Add a node-labels page in RM web UI
- YARN-2940.
Major improvement reported by Varun Saxena and fixed by Li Lu
Fix new findbugs warnings in rest of the hadoop-yarn components
- YARN-2939.
Major improvement reported by Varun Saxena and fixed by Li Lu
Fix new findbugs warnings in hadoop-yarn-common
- YARN-2938.
Major improvement reported by Varun Saxena and fixed by Varun Saxena
Fix new findbugs warnings in hadoop-yarn-resourcemanager and hadoop-yarn-applicationhistoryservice
- YARN-2937.
Major improvement reported by Varun Saxena and fixed by Varun Saxena
Fix new findbugs warnings in hadoop-yarn-nodemanager
- YARN-2936.
Major bug reported by Zhijie Shen and fixed by Varun Saxena
YARNDelegationTokenIdentifier doesn't set proto.builder now
- YARN-2933.
Major sub-task reported by Wangda Tan and fixed by Mayank Bansal (capacityscheduler)
Capacity Scheduler preemption policy should only consider capacity without labels temporarily
- YARN-2932.
Major bug reported by Eric Payne and fixed by Eric Payne
Add entry for "preemptable" status (enabled/disabled) to scheduler web UI and queue initialize/refresh logging
- YARN-2931.
Critical bug reported by Anubhav Dhoot and fixed by Anubhav Dhoot (nodemanager)
PublicLocalizer may fail until directory is initialized by LocalizeRunner
- YARN-2930.
Minor test reported by Ted Yu and fixed by Wangda Tan
TestRMRestart#testRMRestartRecoveringNodeLabelManager sometimes fails against Java 7 & 8
- YARN-2927.
Major sub-task reported by Ray Chiang and fixed by Ray Chiang
InMemorySCMStore properties are inconsistent
- YARN-2924.
Major sub-task reported by Wangda Tan and fixed by Wangda Tan (client)
Node to labels mapping should not transfer to lowercase when adding from RMAdminCLI
- YARN-2922.
Major bug reported by Jason Tufo and fixed by Rohith (capacityscheduler , resourcemanager , scheduler)
ConcurrentModificationException in CapacityScheduler's LeafQueue
- YARN-2920.
Major sub-task reported by Wangda Tan and fixed by Wangda Tan
CapacityScheduler should be notified when labels on nodes changed
- YARN-2917.
Critical bug reported by Rohith and fixed by Rohith (resourcemanager)
Potential deadlock in AsyncDispatcher when system.exit called in AsyncDispatcher#dispatch and AsyscDispatcher#serviceStop from shutdown hook
- YARN-2914.
Minor sub-task reported by Ted Yu and fixed by Varun Saxena
Potential race condition in Singleton implementation of SharedCacheUploaderMetrics, CleanerMetrics, ClientSCMMetrics
- YARN-2912.
Major bug reported by Steve Loughran and fixed by Varun Saxena (test)
Jersey Tests failing with port in use
- YARN-2910.
Major bug reported by Wilfred Spiegelenburg and fixed by Wilfred Spiegelenburg (fairscheduler)
FSLeafQueue can throw ConcurrentModificationException
- YARN-2907.
Trivial bug reported by Rohith and fixed by Rohith (resourcemanager)
SchedulerNode#toString should print all resource detail instead of only memory.
- YARN-2906.
Major bug reported by Jason Lowe and fixed by Jason Lowe (capacityscheduler)
CapacitySchedulerPage shows HTML tags for a queue's Active Users
- YARN-2905.
Blocker bug reported by Jason Lowe and fixed by Varun Saxena
AggregatedLogsBlock page can infinitely loop if the aggregated log file is corrupted
- YARN-2899.
Minor bug reported by Ming Ma and fixed by Ming Ma (nodemanager , test)
Run TestDockerContainerExecutorWithMocks on Linux only
- YARN-2897.
Major bug reported by Mit Desai and fixed by Mit Desai
CrossOriginFilter needs more log statements
- YARN-2894.
Major bug reported by Rohith and fixed by Rohith (resourcemanager)
When ACL's are enabled, if RM switches then application can not be viewed from web.
- YARN-2891.
Minor improvement reported by Dustin Cote and fixed by Dustin Cote (nodemanager)
Failed Container Executor does not provide a clear error message
- YARN-2881.
Major sub-task reported by Anubhav Dhoot and fixed by Anubhav Dhoot (fairscheduler)
Implement PlanFollower for FairScheduler
- YARN-2880.
Major sub-task reported by Wangda Tan and fixed by Rohith (resourcemanager)
Add a test in TestRMRestart to make sure node labels will be recovered if it is enabled
- YARN-2878.
Major improvement reported by Abin Shahab and fixed by Abin Shahab (documentation)
Fix DockerContainerExecutor.apt.vm formatting
- YARN-2874.
Blocker bug reported by Naganarasimha G R and fixed by Naganarasimha G R (resourcemanager)
Dead lock in "DelegationTokenRenewer" which blocks RM to execute any further apps
- YARN-2870.
Trivial bug reported by Masatake Iwasaki and fixed by Masatake Iwasaki (documentation , timelineserver)
Update examples in document of Timeline Server
- YARN-2869.
Major bug reported by Wangda Tan and fixed by Wangda Tan (capacityscheduler , resourcemanager)
CapacityScheduler should trim sub queue names when parse configuration
- YARN-2865.
Critical bug reported by Rohith and fixed by Rohith (resourcemanager)
Application recovery continuously fails with "Application with id already present. Cannot duplicate"
- YARN-2861.
Major bug reported by Zhijie Shen and fixed by Zhijie Shen
Timeline DT secret manager should not reuse the RM's configs.
- YARN-2857.
Critical bug reported by Mohammad Kamrul Islam and fixed by Mohammad Kamrul Islam
ConcurrentModificationException in ContainerLogAppender
- YARN-2856.
Critical bug reported by Rohith and fixed by Rohith (resourcemanager)
Application recovery throw InvalidStateTransitonException: Invalid event: ATTEMPT_KILLED at ACCEPTED
- YARN-2854.
Critical improvement reported by Zhijie Shen and fixed by Naganarasimha G R (timelineserver)
The document about timeline service and generic service needs to be updated
- YARN-2847.
Major bug reported by Jason Lowe and fixed by Olaf Flebbe (nodemanager)
Linux native container executor segfaults if default banned user detected
- YARN-2837.
Blocker new feature reported by Zhijie Shen and fixed by Zhijie Shen (timelineserver)
Timeline server needs to recover the timeline DT when restarting
- YARN-2820.
Major improvement reported by zhihai xu and fixed by zhihai xu (resourcemanager)
Retry in FileSystemRMStateStore when FS's operations fail due to IOException.
- YARN-2816.
Major bug reported by zhihai xu and fixed by zhihai xu (nodemanager)
NM fail to start with NPE during container recovery
- YARN-2815.
Major bug reported by Ferdinand Xu and fixed by Ferdinand Xu
Remove jline from hadoop-yarn-server-common
- YARN-2811.
Major bug reported by Siqi Li and fixed by Siqi Li
In Fair Scheduler, reservation fulfillments shouldn't ignore max share
- YARN-2809.
Major bug reported by Nathan Roberts and fixed by Nathan Roberts (nodemanager)
Implement workaround for linux kernel panic when removing cgroup
- YARN-2808.
Major bug reported by Gordon Wang and fixed by Naganarasimha G R (client)
yarn client tool can not list app_attempt's container info correctly
- YARN-2807.
Minor sub-task reported by Wangda Tan and fixed by Masatake Iwasaki (documentation , resourcemanager)
Option "--forceactive" not works as described in usage of "yarn rmadmin -transitionToActive"
- YARN-2802.
Major improvement reported by zhihai xu and fixed by zhihai xu (resourcemanager)
ClusterMetrics to include AM launch and register delays
- YARN-2800.
Major sub-task reported by Wangda Tan and fixed by Wangda Tan (client , resourcemanager)
Remove MemoryNodeLabelsStore and add a way to enable/disable node labels feature
- YARN-2799.
Minor improvement reported by zhihai xu and fixed by zhihai xu (test)
cleanup TestLogAggregationService based on the change in YARN-90
- YARN-2797.
Minor improvement reported by Karthik Kambatla and fixed by Karthik Kambatla
TestWorkPreservingRMRestart should use ParametrizedSchedulerTestBase
- YARN-2786.
Major sub-task reported by Wangda Tan and fixed by Wangda Tan (api , client , resourcemanager)
Create yarn cluster CLI to enable list node labels collection
- YARN-2780.
Minor improvement reported by Koji Noguchi and fixed by Eric Payne (resourcemanager)
Log aggregated resource allocation in rm-appsummary.log
- YARN-2777.
Major improvement reported by Ted Yu and fixed by Varun Saxena
Mark the end of individual log in aggregated log
- YARN-2766.
Major sub-task reported by Robert Kanter and fixed by Robert Kanter (timelineserver)
ApplicationHistoryManager is expected to return a sorted list of apps/attempts/containers
- YARN-2765.
Major sub-task reported by Jason Lowe and fixed by Jason Lowe (resourcemanager)
Add leveldb-based implementation for RMStateStore
- YARN-2762.
Minor sub-task reported by Rohith and fixed by Rohith (resourcemanager)
RMAdminCLI node-labels-related args should be trimmed and checked before sending to RM
- YARN-2749.
Major bug reported by Xuan Gong and fixed by Xuan Gong
Some testcases from TestLogAggregationService fails in trunk
- YARN-2742.
Minor bug reported by Sangjin Lee and fixed by Wei Yan (fairscheduler)
FairSchedulerConfiguration should allow extra spaces between value and unit
- YARN-2738.
Major sub-task reported by Anubhav Dhoot and fixed by Anubhav Dhoot (fairscheduler)
Add FairReservationSystem for FairScheduler
- YARN-2735.
Trivial bug reported by zhihai xu and fixed by zhihai xu (nodemanager)
diskUtilizationPercentageCutoff and diskUtilizationSpaceCutoff are initialized twice in DirectoryCollection
- YARN-2731.
Major bug reported by Carlo Curino and fixed by Carlo Curino
Fixed RegisterApplicationMasterResponsePBImpl to properly invoke maybeInitBuilder
- YARN-2713.
Major bug reported by Karthik Kambatla and fixed by Karthik Kambatla (resourcemanager)
"RM Home" link in NM should point to one of the RMs in an HA setup
- YARN-2712.
Major sub-task reported by Tsuyoshi Ozawa and fixed by Tsuyoshi Ozawa (resourcemanager)
TestWorkPreservingRMRestart: Augment FS tests with queue and headroom checks
- YARN-2697.
Major bug reported by Zhijie Shen and fixed by haosdent (resourcemanager)
RMAuthenticationHandler is no longer useful
- YARN-2694.
Major sub-task reported by Wangda Tan and fixed by Wangda Tan (capacityscheduler , resourcemanager)
Ensure only single node labels specified in resource request / host, and node label expression only specified when resourceName=ANY
- YARN-2690.
Major sub-task reported by Anubhav Dhoot and fixed by Anubhav Dhoot (fairscheduler)
Make ReservationSystem and its dependent classes independent of Scheduler type
- YARN-2683.
Major sub-task reported by Steve Loughran and fixed by Steve Loughran (api , resourcemanager)
registry config options: document and move to core-default
- YARN-2679.
Major improvement reported by zhihai xu and fixed by zhihai xu (nodemanager)
Add metric for container launch duration
- YARN-2675.
Major bug reported by zhihai xu and fixed by zhihai xu (nodemanager)
containersKilled metrics is not updated when the container is killed during localization
- YARN-2669.
Major improvement reported by Wei Yan and fixed by Wei Yan
FairScheduler: queue names shouldn't allow periods
- YARN-2643.
Trivial improvement reported by Sandy Ryza and fixed by Karthik Kambatla
Don't create a new DominantResourceCalculator on every FairScheduler.allocate call
- YARN-2641.
Major improvement reported by zhihai xu and fixed by zhihai xu (resourcemanager)
Decommission nodes on -refreshNodes instead of next NM-RM heartbeat
- YARN-2637.
Critical bug reported by Wangda Tan and fixed by Craig Welch (resourcemanager)
maximum-am-resource-percent could be respected for both LeafQueue/User when trying to activate applications.
- YARN-2616.
Major sub-task reported by Steve Loughran and fixed by Akshay Radia (client)
Add CLI client to the registry to list, view and manipulate entries
- YARN-2604.
Major improvement reported by Karthik Kambatla and fixed by Robert Kanter (scheduler)
Scheduler should consider max-allocation-* in conjunction with the largest node
- YARN-2574.
Major new feature reported by Subru Krishnan and fixed by Anubhav Dhoot (fairscheduler)
Add support for FairScheduler to the ReservationSystem
- YARN-2543.
Major sub-task reported by Zhijie Shen and fixed by Naganarasimha G R (timelineserver)
Resource usage should be published to the timeline server as well
- YARN-2461.
Minor bug reported by Ray Chiang and fixed by Ray Chiang
Fix PROCFS_USE_SMAPS_BASED_RSS_ENABLED property in YarnConfiguration
- YARN-2432.
Major bug reported by Varun Saxena and fixed by Varun Saxena (resourcemanager)
RMStateStore should process the pending events before close
- YARN-2427.
Major new feature reported by Varun Vasudev and fixed by Varun Vasudev (resourcemanager)
Add support for moving apps between queues in RM web services
- YARN-2414.
Major bug reported by Zhijie Shen and fixed by Wangda Tan (webapp)
RM web UI: app page will crash if app is failed before any attempt has been created
- YARN-2404.
Major sub-task reported by Jian He and fixed by Tsuyoshi Ozawa
Remove ApplicationAttemptState and ApplicationState class in RMStateStore class
- YARN-2375.
Major sub-task reported by Jonathan Eagles and fixed by Mit Desai
Allow enabling/disabling timeline server per framework
- YARN-2360.
Major new feature reported by Ashwin Shankar and fixed by Ashwin Shankar (fairscheduler)
Fair Scheduler: Display dynamic fair share for queues on the scheduler page
- YARN-2356.
Minor bug reported by Sunil G and fixed by Sunil G (client)
yarn status command for non-existent application/application attempt/container is too verbose
- YARN-2340.
Critical bug reported by Nishan Shetty and fixed by Rohith (resourcemanager , scheduler)
NPE thrown when RM restart after queue is STOPPED. There after RM can not recovery application's and remain in standby
- YARN-2315.
Major bug reported by zhihai xu and fixed by zhihai xu
FairScheduler: Set current capacity in addition to capacity
- YARN-2301.
Major improvement reported by Jian He and fixed by Naganarasimha G R
Improve yarn container command
- YARN-2254.
Minor improvement reported by zhihai xu and fixed by zhihai xu
TestRMWebServicesAppsModification should run against both CS and FS
- YARN-2246.
Major bug reported by Devaraj K and fixed by Devaraj K (webapp)
Job History Link in RM UI is redirecting to the URL which contains Job Id twice
- YARN-2243.
Minor bug reported by Ted Yu and fixed by Devaraj K
Order of arguments for Preconditions.checkNotNull() is wrong in SchedulerApplicationAttempt ctor
- YARN-2236.
Major sub-task reported by Chris Trezzo and fixed by Chris Trezzo
Shared Cache uploader service on the Node Manager
- YARN-2230.
Minor bug reported by Adam Kawa and fixed by Vijay Bhat (client , documentation , scheduler)
Fix description of yarn.scheduler.maximum-allocation-vcores in yarn-default.xml (or code)
I have modified the description of the yarn.scheduler.maximum-allocation-vcores setting in yarn-default.xml to be reflective of the actual behavior (throw InvalidRequestException when the limit is crossed).
Since this is a documentation change, I have not added any test cases.
Please review the patch, thanks!
- YARN-2217.
Major sub-task reported by Chris Trezzo and fixed by Chris Trezzo
Shared cache client side changes
- YARN-2213.
Minor task reported by Ted Yu and fixed by Varun Saxena
Change proxy-user cookie log in AmIpFilter to DEBUG
- YARN-2203.
Major sub-task reported by Chris Trezzo and fixed by Chris Trezzo
Web UI for cache manager
- YARN-2190.
Major new feature reported by Chuan Liu and fixed by Chuan Liu (nodemanager)
Add CPU and memory limit options to the default container executor for Windows containers
- YARN-2189.
Major sub-task reported by Chris Trezzo and fixed by Chris Trezzo
Admin service for cache manager
- YARN-2188.
Major sub-task reported by Chris Trezzo and fixed by Chris Trezzo
Client service for cache manager
- YARN-2186.
Major sub-task reported by Chris Trezzo and fixed by Chris Trezzo
Node Manager uploader service for cache manager
- YARN-2183.
Major sub-task reported by Chris Trezzo and fixed by Chris Trezzo
Cleaner service for cache manager
- YARN-2180.
Major sub-task reported by Chris Trezzo and fixed by Chris Trezzo
In-memory backing store for cache manager
- YARN-2179.
Major sub-task reported by Chris Trezzo and fixed by Chris Trezzo
Initial cache manager structure and context
- YARN-2165.
Major sub-task reported by Karam Singh and fixed by Vasanth kumar RJ (timelineserver)
Timeline server should validate the numeric configuration values
- YARN-2157.
Major improvement reported by Akira AJISAKA and fixed by Akira AJISAKA (documentation)
Document YARN metrics
- YARN-2136.
Major bug reported by Jian He and fixed by Varun Saxena
RMStateStore can explicitly handle store/update events when fenced
- YARN-2079.
Major sub-task reported by Jason Lowe and fixed by Jason Lowe (nodemanager)
Recover NonAggregatingLogHandler state upon nodemanager restart
- YARN-2056.
Major sub-task reported by Mayank Bansal and fixed by Eric Payne (resourcemanager)
Disable preemption at Queue level
- YARN-1984.
Major sub-task reported by Jason Lowe and fixed by Varun Saxena
LeveldbTimelineStore does not handle db exceptions properly
- YARN-1979.
Major test reported by Vinod Kumar Vavilapalli and fixed by Vinod Kumar Vavilapalli
TestDirectoryCollection fails when the umask is unusual
- YARN-1904.
Major sub-task reported by Zhijie Shen and fixed by Zhijie Shen
Uniform the XXXXNotFound messages from ClientRMService and ApplicationHistoryClientService
I just committed this. Thanks Zhijie!
- YARN-1884.
Major sub-task reported by Zhijie Shen and fixed by Xuan Gong
ContainerReport should have nodeHttpAddress
- YARN-1809.
Major sub-task reported by Zhijie Shen and fixed by Xuan Gong
Synchronize RM and Generic History Service Web-UIs
- YARN-1723.
Major sub-task reported by Bikas Saha and fixed by Bartosz Ługowski
AMRMClientAsync missing blacklist addition and removal functionality
- YARN-1703.
Critical bug reported by Rohith and fixed by Rohith
Too many connections are opened for proxy server when applicationMaster UI is accessed.
- YARN-1615.
Trivial bug reported by Akira AJISAKA and fixed by Akira AJISAKA (documentation , scheduler)
Fix typos in description about delay scheduling
- YARN-1582.
Major improvement reported by Thomas Graves and fixed by Thomas Graves (capacityscheduler)
Capacity Scheduler: add a maximum-allocation-mb setting per queue
- YARN-1580.
Trivial bug reported by German Florez-Larrahondo and fixed by Brahma Reddy Battula (documentation)
Documentation error regarding "container-allocation.expiry-interval-ms"
- YARN-1537.
Major test reported by Hong Shen and fixed by Xuan Gong (nodemanager)
TestLocalResourcesTrackerImpl.testLocalResourceCache often failed
- YARN-1514.
Major sub-task reported by Tsuyoshi Ozawa and fixed by Tsuyoshi Ozawa
Utility to benchmark ZKRMStateStore#loadState for ResourceManager-HA
- YARN-1453.
Minor bug reported by Andrew Purtell and fixed by Akira AJISAKA
[JDK8] Fix Javadoc errors caused by incorrect or illegal tags in doc comments
- YARN-1393.
Major improvement reported by Wei Yan and fixed by Wei Yan
SLS: Add how-to-use instructions
- YARN-1299.
Major improvement reported by Devaraj K and fixed by (resourcemanager)
Improve a log message in AppSchedulingInfo by adding application id
- YARN-1237.
Minor bug reported by Hitesh Shah and fixed by Brahma Reddy Battula (documentation)
Description for yarn.nodemanager.aux-services in yarn-default.xml is misleading
- YARN-1156.
Major improvement reported by Akira AJISAKA and fixed by Tsuyoshi Ozawa
Enhance NodeManager AllocatedGB and AvailableGB metrics for aggregation of decimal values
- YARN-933.
Major bug reported by J.Andreina and fixed by Rohith (resourcemanager)
Potential InvalidStateTransitonException: Invalid event: LAUNCHED at FINAL_SAVING
- YARN-570.
Major bug reported by Peng Zhang and fixed by Akira AJISAKA (webapp)
Time strings are formated in different timezone
- MAPREDUCE-6303.
Blocker bug reported by Jason Lowe and fixed by Jason Lowe
Read timeout when retrying a fetch error can be fatal to a reducer
- MAPREDUCE-6292.
Minor improvement reported by Akira AJISAKA and fixed by Akira AJISAKA (test)
Use org.junit package instead of junit.framework in TestCombineFileInputFormat
- MAPREDUCE-6285.
Major bug reported by Jonathan Eagles and fixed by Jonathan Eagles
ClientServiceDelegate should not retry upon AuthenticationException
- MAPREDUCE-6277.
Major bug reported by Chang Li and fixed by Chang Li (mr-am)
Job can post multiple history files if attempt loses connection to the RM
- MAPREDUCE-6275.
Critical bug reported by Siqi Li and fixed by Gera Shegalov
Race condition in FileOutputCommitter v2 for user-specified task output subdirs
- MAPREDUCE-6268.
Minor bug reported by Ryu Kobayashi and fixed by Ryu Kobayashi
Fix typo in Task Attempt API's URL
- MAPREDUCE-6267.
Minor improvement reported by Chris Trezzo and fixed by Chris Trezzo
Refactor JobSubmitter#copyAndConfigureFiles into it's own class
- MAPREDUCE-6265.
Major improvement reported by zhihai xu and fixed by zhihai xu (mrv2)
Make ContainerLauncherImpl.INITIAL_POOL_SIZE configurable to better control to launch/kill containers
- MAPREDUCE-6264.
Major task reported by Akira AJISAKA and fixed by Brahma Reddy Battula
Remove httpclient dependency from hadoop-mapreduce-client
- MAPREDUCE-6263.
Major bug reported by Jason Lowe and fixed by Eric Payne (client)
Configurable timeout between YARNRunner terminate the application and forcefully kill.
- MAPREDUCE-6261.
Major bug reported by Jason Lowe and fixed by Tsuyoshi Ozawa (mrv2)
NullPointerException if MapOutputBuffer.flush invoked twice
- MAPREDUCE-6256.
Minor improvement reported by Devaraj K and fixed by Naganarasimha G R
Removed unused private methods in o.a.h.mapreduce.Job.java
- MAPREDUCE-6255.
Minor improvement reported by Ryu Kobayashi and fixed by Ryu Kobayashi (client)
Fix JobCounter's format to use grouping separator
- MAPREDUCE-6253.
Minor improvement reported by Ray Chiang and fixed by Ray Chiang
Update use of Iterator to Iterable
- MAPREDUCE-6248.
Major improvement reported by Jing Zhao and fixed by Jing Zhao (distcp)
Allow users to get the MR job information for distcp
- MAPREDUCE-6243.
Minor bug reported by Akira AJISAKA and fixed by Masatake Iwasaki (tools/rumen)
Fix findbugs warnings in hadoop-rumen
- MAPREDUCE-6233.
Major bug reported by Yongjun Zhang and fixed by zhihai xu (test)
org.apache.hadoop.mapreduce.TestLargeSort.testLargeSort failed in trunk
- MAPREDUCE-6231.
Major bug reported by Akira AJISAKA and fixed by Akira AJISAKA (examples)
Grep example job is not working on a fully-distributed cluster
- MAPREDUCE-6230.
Blocker bug reported by Jason Lowe and fixed by Jason Lowe (mr-am)
MR AM does not survive RM restart if RM activated a new AMRM secret key
- MAPREDUCE-6228.
Major new feature reported by Konstantin Shvachko and fixed by Plamen Jeliazkov (benchmarks , test)
Add truncate operation to SLive
- MAPREDUCE-6227.
Major new feature reported by Konstantin Shvachko and fixed by Konstantin Shvachko (benchmarks , test)
DFSIO for truncate
- MAPREDUCE-6225.
Major bug reported by Jason Lowe and fixed by Varun Saxena
Fix new findbug warnings in hadoop-mapreduce-client-core
- MAPREDUCE-6221.
Minor bug reported by Ted Yu and fixed by Ted Yu
Stringifier is left unclosed in Chain#getChainElementConf()
- MAPREDUCE-6210.
Minor bug reported by Leitao Guo and fixed by Leitao Guo (applicationmaster)
Use getApplicationAttemptId() instead of getApplicationID() for logging AttemptId in RMContainerAllocator.java
- MAPREDUCE-6206.
Critical bug reported by Jens Rabe and fixed by Jens Rabe
TestAggregatedTransferRate fails on non-US systems
- MAPREDUCE-6194.
Minor improvement reported by Harsh J and fixed by Varun Saxena (task)
Bubble up final exception in failures during creation of output collectors
- MAPREDUCE-6186.
Minor bug reported by Rohit Agarwal and fixed by Rohit Agarwal (jobhistoryserver)
Redundant call to requireJob() while displaying the conf page
- MAPREDUCE-6177.
Trivial bug reported by yangping wu and fixed by yangping wu (documentation)
Minor typo in the EncryptedShuffle document about ssl-client.xml
- MAPREDUCE-6173.
Major improvement reported by Junping Du and fixed by Junping Du (distributed-cache , documentation)
Document the configuration of deploying MR over distributed cache with enabling wired encryption at the same time
- MAPREDUCE-6172.
Minor bug reported by Jason Lowe and fixed by Varun Saxena (test)
TestDbClasses timeouts are too aggressive
- MAPREDUCE-6169.
Major improvement reported by zhihai xu and fixed by zhihai xu (mrv2)
MergeQueue should release reference to the current item from key and value at the end of the iteration to save memory.
- MAPREDUCE-6166.
Major bug reported by Eric Payne and fixed by Eric Payne (mrv2)
Reducers do not validate checksum of map outputs when fetching directly to disk
- MAPREDUCE-6162.
Blocker bug reported by Jason Lowe and fixed by Jason Lowe (jobhistoryserver)
mapred hsadmin fails on a secure cluster
- MAPREDUCE-6160.
Major bug reported by Rohith and fixed by Rohith
Potential NullPointerException in MRClientProtocol interface implementation.
- MAPREDUCE-6151.
Minor improvement reported by Masatake Iwasaki and fixed by Masatake Iwasaki (documentation)
Update document of GridMix
- MAPREDUCE-6150.
Minor improvement reported by Masatake Iwasaki and fixed by Masatake Iwasaki (documentation)
Update document of Rumen
- MAPREDUCE-6149.
Major improvement reported by Junping Du and fixed by Junping Du (documentation)
Document override log4j.properties in MR job
- MAPREDUCE-6143.
Major improvement reported by zhihai xu and fixed by zhihai xu (mrv2)
add configuration for mapreduce speculative execution in MR2
- MAPREDUCE-6141.
Major improvement reported by Jason Lowe and fixed by Jason Lowe (jobhistoryserver)
History server leveldb recovery store
- MAPREDUCE-6136.
Major bug reported by Noah Watkins and fixed by Brahma Reddy Battula (applicationmaster)
MRAppMaster doesn't shutdown file systems
- MAPREDUCE-6059.
Major improvement reported by Siqi Li and fixed by Siqi Li
Speed up history server startup time
- MAPREDUCE-6049.
Major bug reported by Nishan Shetty and fixed by Rohith (applicationmaster , resourcemanager)
AM JVM does not exit if MRClientService gracefull shutdown fails
- MAPREDUCE-6046.
Minor improvement reported by Devaraj K and fixed by Sahil Takiar (mr-am)
Change the class name for logs in RMCommunicator.java
- MAPREDUCE-6045.
Minor bug reported by zhihai xu and fixed by zhihai xu (test)
need close the DataInputStream after open it in TestMapReduce.java
- MAPREDUCE-5988.
Minor bug reported by Akira AJISAKA and fixed by Akira AJISAKA (documentation)
Fix dead links to the javadocs in mapreduce project
- MAPREDUCE-5932.
Major improvement reported by Gera Shegalov and fixed by Gera Shegalov (mrv2)
Provide an option to use a dedicated reduce-side shuffle log
- MAPREDUCE-5918.
Major bug reported by Sergey Murylev and fixed by Sergey Murylev
LineRecordReader can return the same decompressor to CodecPool multiple times
- MAPREDUCE-5800.
Minor improvement reported by Akira AJISAKA and fixed by Akira AJISAKA
Use Job#getInstance instead of deprecated constructors
- MAPREDUCE-5657.
Minor bug reported by Andrew Purtell and fixed by Akira AJISAKA (documentation)
[JDK8] Fix Javadoc errors caused by incorrect or illegal tags in doc comments
- MAPREDUCE-5612.
Minor improvement reported by Sandy Ryza and fixed by Chris Palmer (documentation)
Add javadoc for TaskCompletionEvent.Status
- MAPREDUCE-5583.
Major improvement reported by Jason Lowe and fixed by Jason Lowe (mr-am , mrv2)
Ability to limit running map and reduce tasks
- MAPREDUCE-5568.
Major bug reported by Jian He and fixed by MinJi Kim
JHS returns invalid string for reducer completion percentage if AM restarts with 0 reducer.
- MAPREDUCE-5420.
Major task reported by Sandy Ryza and fixed by James Carman
Remove mapreduce.task.tmp.dir from mapred-default.xml
- MAPREDUCE-5335.
Major improvement reported by Devaraj K and fixed by Devaraj K (applicationmaster)
Rename Job Tracker terminology in ShuffleSchedulerImpl
- MAPREDUCE-4879.
Major bug reported by Gera Shegalov and fixed by Gera Shegalov (examples)
TeraOutputFormat may overwrite an existing output directory
- MAPREDUCE-4815.
Major improvement reported by Jason Lowe and fixed by Siqi Li (mrv2)
Speed up FileOutputCommitter#commitJob for many output files
- MAPREDUCE-4742.
Trivial bug reported by Liang Xie and fixed by Liang Xie (test)
Fix typo in nnbench#displayUsage
- MAPREDUCE-4431.
Minor improvement reported by Nishan Shetty and fixed by Devaraj K (mrv2)
mapred command should print the reason on killing already completed jobs
- MAPREDUCE-4286.
Major bug reported by Devaraj K and fixed by Devaraj K
TestClientProtocolProviderImpls passes on failure conditions
- MAPREDUCE-3283.
Minor bug reported by Ramya Sunil and fixed by Varun Saxena (scripts)
mapred classpath CLI does not display the complete classpath
- MAPREDUCE-2815.
Minor bug reported by Shane Butler and fixed by Chris Palmer (documentation)
JavaDoc does not generate correctly for MultithreadedMapRunner
- HDFS-8072.
Major bug reported by Arpit Agarwal and fixed by Arpit Agarwal (datanode)
Reserved RBW space is not released if client terminates while writing block
- HDFS-8071.
Major improvement reported by Konstantin Shvachko and fixed by Konstantin Shvachko
Redundant checkFileProgress() in PART II of getAdditionalBlock()
- HDFS-8063.
Minor bug reported by Masatake Iwasaki and fixed by Masatake Iwasaki (test)
Fix intermittent test failures in TestTracing
- HDFS-8051.
Major bug reported by Lei (Eddy) Xu and fixed by Lei (Eddy) Xu (datanode)
FsVolumeList#addVolume should release volume reference if not put it into BlockScanner.
- HDFS-8039.
Minor sub-task reported by Xiaoyu Yao and fixed by Xiaoyu Yao (test)
Fix TestDebugAdmin#testRecoverLease and testVerfiyBlockChecksumCommand on Windows
- HDFS-8038.
Minor sub-task reported by Xiaoyu Yao and fixed by Xiaoyu Yao (tools)
PBImageDelimitedTextWriter#getEntry output HDFS path in platform-specific format.
- HDFS-8036.
Major bug reported by Jing Zhao and fixed by Jing Zhao (distcp)
Use snapshot path as source when using snapshot diff report in DistCp
- HDFS-8025.
Major bug reported by Ming Ma and fixed by Ming Ma
Addendum fix for HDFS-3087 Decomissioning on NN restart can complete without blocks being replicated
- HDFS-8001.
Trivial bug reported by Remi Catherinot and fixed by Remi Catherinot (nfs)
RpcProgramNfs3 : wrong parsing of dfs.blocksize
patch is fully backward compatible.
- HDFS-7999.
Major bug reported by zhouyingchao and fixed by zhouyingchao
FsDatasetImpl#createTemporary sometimes holds the FSDatasetImpl lock for a very long time
- HDFS-7996.
Critical bug reported by Lei (Eddy) Xu and fixed by Lei (Eddy) Xu (datanode)
After swapping a volume, BlockReceiver reports ReplicaNotFoundException
- HDFS-7977.
Major bug reported by Brandon Li and fixed by Brandon Li (nfs)
NFS couldn't take percentile intervals
- HDFS-7976.
Major improvement reported by Brandon Li and fixed by Brandon Li (documentation , nfs)
Update NFS user guide for mount option "sync" to minimize or avoid reordered writes
- HDFS-7963.
Critical bug reported by Masatake Iwasaki and fixed by Masatake Iwasaki (test)
Fix expected tracing spans in TestTracing along with HDFS-7054
- HDFS-7962.
Minor improvement reported by Yi Liu and fixed by Yi Liu
Remove duplicated logs in BlockManager
- HDFS-7961.
Major bug reported by Lei (Eddy) Xu and fixed by Lei (Eddy) Xu
Trigger full block report after hot swapping disk
- HDFS-7960.
Critical bug reported by Lei (Eddy) Xu and fixed by Colin Patrick McCabe
The full block report should prune zombie storages even if they're not empty
- HDFS-7957.
Critical bug reported by Jing Zhao and fixed by Jing Zhao (namenode)
Truncate should verify quota before making changes
- HDFS-7956.
Major bug reported by Konstantin Shvachko and fixed by Plamen Jeliazkov (namenode)
Improve logging for DatanodeRegistration.
- HDFS-7954.
Minor sub-task reported by Xiaoyu Yao and fixed by Xiaoyu Yao (test)
TestBalancer#testBalancerWithPinnedBlocks failed on Windows
- HDFS-7953.
Minor bug reported by kanaka kumar avvaru and fixed by kanaka kumar avvaru (namenode)
NN Web UI fails to navigate to paths that contain #
- HDFS-7951.
Major sub-task reported by Xiaoyu Yao and fixed by Xiaoyu Yao (test)
Fix NPE for TestFsDatasetImpl#testAddVolumeFailureReleasesInUseLock on Linux
- HDFS-7950.
Major sub-task reported by Xiaoyu Yao and fixed by Xiaoyu Yao (test)
Fix TestFsDatasetImpl#testAddVolumes failure on Windows
- HDFS-7948.
Major sub-task reported by Xiaoyu Yao and fixed by Xiaoyu Yao (test)
TestDataNodeHotSwapVolumes#testAddVolumeFailures failed on Windows
- HDFS-7946.
Major sub-task reported by Xiaoyu Yao and fixed by Xiaoyu Yao (test)
TestDataNodeVolumeFailureReporting NPE on Windows
- HDFS-7945.
Blocker bug reported by Haohui Mai and fixed by Haohui Mai
The WebHdfs system on DN does not honor the length parameter
- HDFS-7943.
Blocker bug reported by Jing Zhao and fixed by Jing Zhao
Append cannot handle the last block with length greater than the preferred block size
- HDFS-7942.
Major bug reported by Brandon Li and fixed by Brandon Li (nfs)
NFS: support regexp grouping in nfs.exports.allowed.hosts
- HDFS-7940.
Major sub-task reported by Rakesh R and fixed by Rakesh R (hdfs-client)
Add tracing to DFSClient#setQuotaByStorageType
- HDFS-7932.
Major bug reported by Kihwal Lee and fixed by Kihwal Lee
Speed up the shutdown of datanode during rolling upgrade
- HDFS-7930.
Blocker bug reported by Konstantin Shvachko and fixed by Yi Liu (namenode)
commitBlockSynchronization() does not remove locations
- HDFS-7929.
Major bug reported by Zhe Zhang and fixed by Zhe Zhang
inotify unable fetch pre-upgrade edit log segments once upgrade starts
- HDFS-7926.
Major bug reported by Tsz Wo Nicholas Sze and fixed by Tsz Wo Nicholas Sze (namenode)
NameNode implementation of ClientProtocol.truncate(..) is not idempotent
- HDFS-7917.
Minor improvement reported by Lei (Eddy) Xu and fixed by Lei (Eddy) Xu (test)
Use file to replace data dirs in test to simulate a disk failure.
- HDFS-7915.
Major bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe
The DataNode can sometimes allocate a ShortCircuitShm slot and fail to tell the DFSClient about it because of a network error
- HDFS-7914.
Major test reported by Xiaoyu Yao and fixed by Xiaoyu Yao
TestJournalNode#testFailToStartWithBadConfig fails when the default dfs.journalnode.http-address port 8480 is in use
- HDFS-7903.
Blocker sub-task reported by Tsz Wo Nicholas Sze and fixed by Plamen Jeliazkov (datanode , namenode)
Cannot recover block after truncate and delete snapshot
- HDFS-7898.
Minor improvement reported by Tsz Wo Nicholas Sze and fixed by Tsz Wo Nicholas Sze (test)
Change TestAppendSnapshotTruncate to fail-fast
- HDFS-7886.
Minor bug reported by Yi Liu and fixed by Plamen Jeliazkov (test)
TestFileTruncate#testTruncateWithDataNodesRestart runs timeout sometimes
- HDFS-7885.
Critical bug reported by vitthal (Suhas) Gogate and fixed by Tsz Wo Nicholas Sze (datanode)
Datanode should not trust the generation stamp provided by client
- HDFS-7884.
Blocker bug reported by Tsz Wo Nicholas Sze and fixed by Brahma Reddy Battula (datanode)
NullPointerException in BlockSender
- HDFS-7881.
Blocker bug reported by Akira AJISAKA and fixed by Brahma Reddy Battula
TestHftpFileSystem#testSeek fails in branch-2
- HDFS-7880.
Blocker bug reported by Akira AJISAKA and fixed by Brahma Reddy Battula (test)
Remove the tests for legacy Web UI in branch-2
- HDFS-7879.
Major bug reported by Chris Nauroth and fixed by Chris Nauroth (build , libhdfs)
hdfs.dll does not export functions of the public libhdfs API
- HDFS-7871.
Critical bug reported by Jing Zhao and fixed by Jing Zhao
NameNodeEditLogRoller can keep printing "Swallowing exception" message
- HDFS-7869.
Major bug reported by J.Andreina and fixed by J.Andreina
Inconsistency in the return information while performing rolling upgrade
- HDFS-7855.
Major sub-task reported by Li Bo and fixed by Li Bo (hdfs-client)
Separate class Packet from DFSOutputStream
- HDFS-7849.
Minor improvement reported by Tsz Wo Nicholas Sze and fixed by J.Andreina (documentation)
Update documentation for enabling a new feature in rolling upgrade
- HDFS-7843.
Blocker sub-task reported by Tsz Wo Nicholas Sze and fixed by Tsz Wo Nicholas Sze (namenode)
A truncated file is corrupted after rollback from a rolling upgrade
- HDFS-7838.
Major sub-task reported by Yi Liu and fixed by Yi Liu (datanode , namenode)
Expose truncate API for libhdfs
- HDFS-7832.
Major improvement reported by Vinayakumar B and fixed by Vinayakumar B (namenode)
Show 'Last Modified' in Namenode's 'Browse Filesystem'
- HDFS-7831.
Major bug reported by Konstantin Shvachko and fixed by Konstantin Shvachko
Fix the starting index and end condition of the loop in FileDiffList.findEarlierSnapshotBlocks()
- HDFS-7830.
Major bug reported by Lei (Eddy) Xu and fixed by Lei (Eddy) Xu (datanode)
DataNode does not release the volume lock when adding a volume fails.
- HDFS-7824.
Major sub-task reported by Xiaoyu Yao and fixed by Xiaoyu Yao (datanode , namenode)
GetContentSummary API and its namenode implementation for Storage Type Quota/Usage
- HDFS-7819.
Major sub-task reported by Rakesh R and fixed by Rakesh R (datanode)
Log WARN message for the blocks which are not in Block ID based layout
- HDFS-7818.
Blocker bug reported by Eric Payne and fixed by Eric Payne (webhdfs)
OffsetParam should return the default value instead of throwing NPE when the value is unspecified
- HDFS-7816.
Blocker bug reported by Jason Lowe and fixed by Haohui Mai (webhdfs)
Unable to open webhdfs paths with "+"
- HDFS-7814.
Minor sub-task reported by Xiaoyu Yao and fixed by Xiaoyu Yao (datanode , namenode)
Fix usage string of storageType parameter for "dfsadmin -setSpaceQuota/clrSpaceQuota"
- HDFS-7813.
Major bug reported by Rakesh R and fixed by Rakesh R (ha , test)
TestDFSHAAdminMiniCluster#testFencer testcase is failing frequently
- HDFS-7811.
Minor sub-task reported by Xiaoyu Yao and fixed by Xiaoyu Yao (namenode)
Avoid recursive call getStoragePolicyID in INodeFile#computeQuotaUsage
- HDFS-7807.
Major bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (native)
libhdfs htable.c: fix htable resizing, add unit test
- HDFS-7806.
Minor sub-task reported by Xiaoyu Yao and fixed by Xiaoyu Yao (datanode , namenode)
Refactor: move StorageType from hadoop-hdfs to hadoop-common
This fix moves the public class StorageType from the package org.apache.hadoop.hdfs to org.apache.hadoop.fs.
- HDFS-7805.
Major bug reported by surendra singh lilhore and fixed by surendra singh lilhore (namenode)
NameNode recovery prompt should be printed on console
- HDFS-7798.
Critical bug reported by Chengbing Liu and fixed by Chengbing Liu (security)
Checkpointing failure caused by shared KerberosAuthenticator
- HDFS-7797.
Major improvement reported by Rakesh R and fixed by Rakesh R (namenode)
Add audit log for setQuota operation
- HDFS-7795.
Minor improvement reported by Kihwal Lee and fixed by Kihwal Lee
Show warning if not all favored nodes were chosen by namenode
- HDFS-7790.
Minor improvement reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (hdfs-client)
Do not create optional fields in DFSInputStream unless they are needed
- HDFS-7789.
Major improvement reported by Gera Shegalov and fixed by Gera Shegalov (tools)
DFSck should resolve the path to support cross-FS symlinks
- HDFS-7788.
Blocker bug reported by Kihwal Lee and fixed by Rushabh S Shah
Post-2.6 namenode may not start up with an image containing inodes created with an old release.
- HDFS-7785.
Major bug reported by Chengbing Liu and fixed by Chengbing Liu (namenode)
Improve diagnostics information for HttpPutFailedException
- HDFS-7780.
Minor improvement reported by Ray Chiang and fixed by Ray Chiang
Update use of Iterator to Iterable in DataXceiverServer and SnapshotDiffInfo
- HDFS-7778.
Major bug reported by Lei (Eddy) Xu and fixed by Lei (Eddy) Xu (datanode , test)
Rename FsVolumeListTest to TestFsVolumeList and commit it to branch-2
- HDFS-7776.
Major sub-task reported by Xiaoyu Yao and fixed by Xiaoyu Yao (test)
Adding additional unit tests for Quota By Storage Type
- HDFS-7775.
Minor sub-task reported by Xiaoyu Yao and fixed by Xiaoyu Yao (datanode , namenode)
Use consistent naming for NN-internal quota related types and functions
- HDFS-7774.
Critical bug reported by Venkatasubramaniam Ramakrishnan and fixed by Kiran Kumar M R (build , native)
Unresolved symbols error while compiling HDFS on Windows 7/32 bit
LibHDFS now supports 32-bit build targets on Windows.
- HDFS-7773.
Major improvement reported by Anu Engineer and fixed by Anu Engineer (datanode , namenode)
Additional metrics in HDFS to be accessed via jmx.
- HDFS-7772.
Trivial improvement reported by Xiaoyu Yao and fixed by Xiaoyu Yao (documentation)
Document hdfs balancer -exclude/-include option in HDFSCommands.html
- HDFS-7771.
Major improvement reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe
fuse_dfs should permit FILE: on the front of KRB5CCNAME
- HDFS-7769.
Trivial bug reported by Tsz Wo Nicholas Sze and fixed by (test)
TestHDFSCLI create files in hdfs project root dir
- HDFS-7763.
Major bug reported by Liang Xie and fixed by Liang Xie (ha)
fix zkfc hung issue due to not catching exception in a corner case
- HDFS-7761.
Minor improvement reported by Yi Liu and fixed by Yi Liu
cleanup unnecssary code logic in LocatedBlock
- HDFS-7760.
Minor sub-task reported by Yi Liu and fixed by Konstantin Shvachko (documentation)
Document truncate for WebHDFS.
- HDFS-7757.
Major improvement reported by Arpit Agarwal and fixed by Brahma Reddy Battula (namenode)
Misleading error messages in FSImage.java
- HDFS-7756.
Major bug reported by Ted Yu and fixed by Ted Yu
Restore method signature for LocatedBlock#getLocations()
- HDFS-7753.
Major bug reported by Rakesh R and fixed by Konstantin Shvachko
Fix Multithreaded correctness Warnings in BackupImage.java
- HDFS-7752.
Minor improvement reported by Wellington Chevreuil and fixed by Wellington Chevreuil (documentation)
Improve description for "dfs.namenode.num.extra.edits.retained" and "dfs.namenode.num.checkpoints.retained" properties on hdfs-default.xml
- HDFS-7748.
Blocker bug reported by Haohui Mai and fixed by Anu Engineer
Separate ECN flags from the Status in the DataTransferPipelineAck
- HDFS-7746.
Minor sub-task reported by Tsz Wo Nicholas Sze and fixed by Tsz Wo Nicholas Sze (test)
Add a test randomly mixing append, truncate and snapshot
- HDFS-7744.
Major bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (hdfs-client)
Fix potential NPE in DFSInputStream after setDropBehind or setReadahead is called
- HDFS-7743.
Minor improvement reported by Jing Zhao and fixed by Jing Zhao (namenode)
Code cleanup of BlockInfo and rename BlockInfo to BlockInfoContiguous
- HDFS-7742.
Major bug reported by Nathan Roberts and fixed by Nathan Roberts (namenode)
favoring decommissioning node for replication can cause a block to stay underreplicated for long periods
- HDFS-7741.
Minor bug reported by Yi Liu and fixed by Yi Liu
Remove unnecessary synchronized in FSDataInputStream and HdfsDataInputStream
- HDFS-7740.
Major sub-task reported by Konstantin Shvachko and fixed by Yi Liu (test)
Test truncate with DataNodes restarting
- HDFS-7738.
Minor sub-task reported by Tsz Wo Nicholas Sze and fixed by Tsz Wo Nicholas Sze (test)
Add more tests for truncate
- HDFS-7734.
Blocker bug reported by Arpit Agarwal and fixed by Yi Liu (namenode)
Class cast exception in NameNode#main
- HDFS-7732.
Trivial improvement reported by Akira AJISAKA and fixed by Brahma Reddy Battula
Fix the order of the parameters in DFSConfigKeys
- HDFS-7723.
Major sub-task reported by Xiaoyu Yao and fixed by Xiaoyu Yao (namenode)
Quota By Storage Type namenode implemenation
- HDFS-7722.
Major bug reported by Lei (Eddy) Xu and fixed by Lei (Eddy) Xu (datanode)
DataNode#checkDiskError should also remove Storage when error is found.
- HDFS-7721.
Major bug reported by Tsz Wo Nicholas Sze and fixed by Colin Patrick McCabe (datanode)
The HDFS BlockScanner may run fast during the first hour
- HDFS-7720.
Major sub-task reported by Xiaoyu Yao and fixed by Xiaoyu Yao (datanode , namenode)
Quota by Storage Type API, tools and ClientNameNode Protocol changes
- HDFS-7719.
Major bug reported by Lei (Eddy) Xu and fixed by Lei (Eddy) Xu
BlockPoolSliceStorage#removeVolumes fails to remove some in-memory state associated with volumes
- HDFS-7718.
Major bug reported by Arun Suresh and fixed by Arun Suresh
Store KeyProvider in ClientContext to avoid leaking key provider threads when using FileContext
- HDFS-7714.
Major bug reported by Chris Nauroth and fixed by Vinayakumar B (datanode)
Simultaneous restart of HA NameNodes and DataNode can cause DataNode to register successfully with only one NameNode.
- HDFS-7710.
Minor improvement reported by Xiaoyu Yao and fixed by Xiaoyu Yao
Remove dead code in BackupImage.java
- HDFS-7709.
Major bug reported by Rakesh R and fixed by Rakesh R
Fix findbug warnings in httpfs
- HDFS-7707.
Major bug reported by Yongjun Zhang and fixed by Yongjun Zhang (namenode)
Edit log corruption due to delayed block removal again
- HDFS-7706.
Minor improvement reported by Andrew Wang and fixed by Andrew Wang (namenode)
Switch BlockManager logging to use slf4j
- HDFS-7704.
Major bug reported by Rushabh S Shah and fixed by Rushabh S Shah (datanode , namenode)
DN heartbeat to Active NN may be blocked and expire if connection to Standby NN continues to time out.
- HDFS-7703.
Major improvement reported by Vinayakumar B and fixed by Vinayakumar B
Support favouredNodes for the append for new blocks
- HDFS-7700.
Major sub-task reported by Xiaoyu Yao and fixed by Xiaoyu Yao (documentation)
Document quota support for storage types
- HDFS-7698.
Major bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe
Fix locking on HDFS read statistics and add a method for clearing them.
- HDFS-7697.
Major bug reported by Haohui Mai and fixed by Lei (Eddy) Xu
Mark the PB OIV tool as experimental
- HDFS-7696.
Minor bug reported by Tsz Wo Nicholas Sze and fixed by Tsz Wo Nicholas Sze (datanode)
FsDatasetImpl.getTmpInputStreams(..) may leak file descriptors
- HDFS-7694.
Major improvement reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe
FSDataInputStream should support "unbuffer"
- HDFS-7686.
Blocker bug reported by Rushabh S Shah and fixed by Colin Patrick McCabe
Re-add rapid rescan of possibly corrupt block feature to the block scanner
- HDFS-7685.
Minor improvement reported by Frank Lanitz and fixed by Kai Sasaki (documentation)
Document dfs.namenode.heartbeat.recheck-interval in hdfs-default.xml
- HDFS-7684.
Major improvement reported by Tianyin Xu and fixed by Anu Engineer
The host:port settings of the daemons should be trimmed before use
- HDFS-7683.
Minor improvement reported by Vinayakumar B and fixed by Vinayakumar B (namenode)
Combine usages and percent stats in NameNode UI
- HDFS-7682.
Major bug reported by Charles Lamb and fixed by Charles Lamb
{{DistributedFileSystem#getFileChecksum}} of a snapshotted file includes non-snapshotted content
- HDFS-7681.
Major sub-task reported by Joe Pallas and fixed by Joe Pallas (datanode)
Fix ReplicaInputStream constructor to take InputStreams
- HDFS-7677.
Major sub-task reported by Yi Liu and fixed by Yi Liu (datanode , namenode)
DistributedFileSystem#truncate should resolve symlinks
- HDFS-7676.
Major sub-task reported by Konstantin Shvachko and fixed by Konstantin Shvachko (test)
Fix TestFileTruncate to avoid bug of HDFS-7611
- HDFS-7675.
Trivial improvement reported by Konstantin Shvachko and fixed by Colin Patrick McCabe (hdfs-client)
Remove unused member DFSClient#spanReceiverHost
- HDFS-7668.
Major improvement reported by Allen Wittenauer and fixed by Masatake Iwasaki (documentation)
Convert site documentation from apt to markdown
- HDFS-7660.
Minor bug reported by Lei (Eddy) Xu and fixed by Lei (Eddy) Xu
BlockReceiver#close() might be called multiple times, which causes the fsvolume reference being released incorrectly.
- HDFS-7659.
Major sub-task reported by Yi Liu and fixed by Yi Liu (datanode , namenode)
We should check the new length of truncate can't be a negative value.
- HDFS-7656.
Major sub-task reported by Yi Liu and fixed by Yi Liu (datanode , namenode)
Expose truncate API for HDFS httpfs
- HDFS-7655.
Major sub-task reported by Yi Liu and fixed by Yi Liu (datanode , namenode)
Expose truncate API for Web HDFS
- HDFS-7647.
Major bug reported by Milan Desai and fixed by Milan Desai
DatanodeManager.sortLocatedBlocks sorts DatanodeInfos but not StorageIDs
- HDFS-7644.
Trivial bug reported by Charles Lamb and fixed by Charles Lamb (documentation)
minor typo in HttpFS doc
- HDFS-7643.
Major sub-task reported by Arpit Agarwal and fixed by Yi Liu (test)
Test case to ensure lazy persist files cannot be truncated
- HDFS-7641.
Minor bug reported by Yi Liu and fixed by Yi Liu (documentation)
Update archival storage user doc for list/set/get block storage policies
- HDFS-7640.
Trivial improvement reported by Brandon Li and fixed by Brandon Li (nfs)
print NFS Client in the NFS log
- HDFS-7638.
Major sub-task reported by Yi Liu and fixed by Yi Liu (datanode , namenode)
Small fix and few refinements for FSN#truncate
- HDFS-7637.
Minor bug reported by Yi Liu and fixed by Yi Liu
Fix the check condition for reserved path
- HDFS-7635.
Minor bug reported by Chris Nauroth and fixed by Chris Nauroth (test)
Remove TestCorruptFilesJsp from branch-2.
- HDFS-7634.
Major sub-task reported by Yi Liu and fixed by Yi Liu (datanode , namenode)
Disallow truncation of Lazy persist files
- HDFS-7632.
Major bug reported by Chris Nauroth and fixed by Chris Nauroth (test)
MiniDFSCluster configures DataNode data directories incorrectly if using more than 1 DataNode and more than 2 storage locations per DataNode.
- HDFS-7623.
Major sub-task reported by Yi Liu and fixed by Yi Liu
Add htrace configuration properties to core-default.xml and update user doc about how to enable htrace
- HDFS-7615.
Major bug reported by Kihwal Lee and fixed by Kihwal Lee
Remove longReadLock
- HDFS-7611.
Critical bug reported by Konstantin Shvachko and fixed by Jing Zhao (namenode)
deleteSnapshot and delete of a file can leave orphaned blocks in the blocksMap on NameNode restart.
- HDFS-7610.
Major bug reported by Lei (Eddy) Xu and fixed by Lei (Eddy) Xu (datanode)
Fix removal of dynamically added DN volumes
- HDFS-7606.
Minor bug reported by Ted Yu and fixed by Byron Wong
Missing null check in INodeFile#getBlocks()
- HDFS-7604.
Major improvement reported by Chris Nauroth and fixed by Chris Nauroth (datanode , namenode)
Track and display failed DataNode storage locations in NameNode.
- HDFS-7603.
Critical bug reported by Kihwal Lee and fixed by Kihwal Lee (rolling upgrades)
The background replication queue initialization may not let others run
- HDFS-7600.
Major improvement reported by Yi Liu and fixed by Jing Zhao (tools)
Refine hdfs admin classes to reuse common code
- HDFS-7598.
Minor improvement reported by Sangjin Lee and fixed by Sangjin Lee (test)
Remove dependency on old version of Guava in TestDFSClientCache#testEviction
- HDFS-7596.
Major bug reported by Arpit Agarwal and fixed by Arpit Agarwal (namenode)
NameNode should prune dead storages from storageMap
- HDFS-7591.
Minor improvement reported by Chris Nauroth and fixed by Varun Saxena (scripts)
hdfs classpath command should support same options as hadoop classpath.
- HDFS-7589.
Major sub-task reported by Zhanwei Wang and fixed by Zhanwei Wang (libhdfs)
Break the dependency between libnative_mini_dfs and libhdfs
- HDFS-7587.
Blocker bug reported by Kihwal Lee and fixed by Jing Zhao (namenode)
Edit log corruption can happen if append fails with a quota violation
- HDFS-7585.
Major test reported by sam liu and fixed by sam liu (test)
Get TestEnhancedByteBufferAccess working on CPU architectures with page sizes other than 4096
- HDFS-7584.
Major new feature reported by Xiaoyu Yao and fixed by Xiaoyu Yao (datanode , namenode)
Enable Quota Support for Storage Types
1. Introduced quota by storage type as a hard limit on the amount of space usage allowed for different storage types (SSD, DISK, ARCHIVE) under the target directory.
2. Added {{SetQuotaByStorageType}} API and {{-storagetype}} option for {{hdfs dfsadmin -setSpaceQuota/-clrSpaceQuota}} commands to allow set/clear quota by storage type under the target directory.
- HDFS-7583.
Minor bug reported by Vinayakumar B and fixed by Vinayakumar B (namenode)
Fix findbug in TransferFsImage.java
- HDFS-7579.
Minor improvement reported by Charles Lamb and fixed by Charles Lamb (datanode)
Improve log reporting during block report rpc failure
- HDFS-7575.
Critical bug reported by Lars Francke and fixed by Arpit Agarwal
Upgrade should generate a unique storage ID for each volume
- HDFS-7573.
Major sub-task reported by Haohui Mai and fixed by Haohui Mai
Consolidate the implementation of delete() into a single class
- HDFS-7572.
Major bug reported by Arpit Agarwal and fixed by Arpit Agarwal (test)
TestLazyPersistFiles#testDnRestartWithSavedReplicas is flaky on Windows
- HDFS-7566.
Major bug reported by Ray Chiang and fixed by Ray Chiang
Remove obsolete entries from hdfs-default.xml
- HDFS-7564.
Minor improvement reported by Hari Sekhon and fixed by Yongjun Zhang (nfs)
NFS gateway dynamically reload UID/GID mapping file /etc/nfs.map
- HDFS-7563.
Major bug reported by Hari Sekhon and fixed by Yongjun Zhang (nfs)
NFS gateway parseStaticMap NumberFormatException
- HDFS-7561.
Major bug reported by Konstantin Shvachko and fixed by Liang Xie
TestFetchImage should write fetched-image-dir under target.
- HDFS-7560.
Critical bug reported by Vinayakumar B and fixed by Vinayakumar B (namenode)
ACLs removed by removeDefaultAcl() will be back after NameNode restart/failover
- HDFS-7557.
Minor improvement reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe
Fix spacing for a few keys in DFSConfigKeys.java
- HDFS-7555.
Major improvement reported by Haohui Mai and fixed by Haohui Mai
Remove the support of unmanaged connectors in HttpServer2
- HDFS-7552.
Major bug reported by Liang Xie and fixed by Liang Xie (datanode , test)
change FsVolumeList toString() to fix TestDataNodeVolumeFailureToleration
- HDFS-7548.
Major bug reported by Rushabh S Shah and fixed by Rushabh S Shah
Corrupt block reporting delayed until datablock scanner thread detects it
- HDFS-7543.
Major sub-task reported by Haohui Mai and fixed by Haohui Mai
Avoid path resolution when getting FileStatus for audit logs
- HDFS-7537.
Major improvement reported by Allen Wittenauer and fixed by GAO Rui (namenode)
fsck is confusing when dfs.namenode.replication.min > 1 && missing replicas && NN restart
- HDFS-7536.
Minor bug reported by Yi Liu and fixed by Yi Liu (security)
Remove unused CryptoCodec in org.apache.hadoop.fs.Hdfs
- HDFS-7535.
Major improvement reported by Jing Zhao and fixed by Jing Zhao (distcp , snapshots)
Utilize Snapshot diff report for distcp
- HDFS-7533.
Major bug reported by Kihwal Lee and fixed by Eric Payne
Datanode sometimes does not shutdown on receiving upgrade shutdown command
- HDFS-7531.
Major improvement reported by Lei (Eddy) Xu and fixed by Lei (Eddy) Xu (datanode)
Improve the concurrent access on FsVolumeList
- HDFS-7530.
Minor bug reported by Charles Lamb and fixed by Charles Lamb (namenode)
Allow renaming of encryption zone roots
- HDFS-7528.
Major sub-task reported by Haohui Mai and fixed by Haohui Mai
Consolidate symlink-related implementation into a single class
- HDFS-7517.
Major bug reported by Haohui Mai and fixed by Haohui Mai
Remove redundant non-null checks in FSNamesystem#getBlockLocations
- HDFS-7516.
Major bug reported by Brandon Li and fixed by Brandon Li (nfs)
Fix findbugs warnings in hadoop-nfs project
- HDFS-7515.
Major bug reported by Haohui Mai and fixed by Haohui Mai
Fix new findbugs warnings in hadoop-hdfs
- HDFS-7514.
Major bug reported by Arpit Agarwal and fixed by Arpit Agarwal (test)
TestTextCommand fails on Windows
- HDFS-7513.
Major improvement reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (namenode)
HDFS inotify: add defaultBlockSize to CreateEvent
- HDFS-7509.
Major sub-task reported by Jing Zhao and fixed by Jing Zhao
Avoid resolving path multiple times
- HDFS-7506.
Major sub-task reported by Haohui Mai and fixed by Haohui Mai
Consolidate implementation of setting inode attributes into a single class
- HDFS-7502.
Major bug reported by Brandon Li and fixed by Brandon Li (nfs)
Fix findbugs warning in hdfs-nfs project
- HDFS-7498.
Major sub-task reported by Jing Zhao and fixed by Jing Zhao (namenode)
Simplify the logic in INodesInPath
- HDFS-7497.
Major bug reported by Yongjun Zhang and fixed by Yongjun Zhang (datanode , namenode)
Inconsistent report of decommissioning DataNodes between dfsadmin and NameNode webui
- HDFS-7496.
Major bug reported by Colin Patrick McCabe and fixed by Lei (Eddy) Xu
Fix FsVolume removal race conditions on the DataNode by reference-counting the volume instances
- HDFS-7495.
Minor bug reported by Ted Yu and fixed by Colin Patrick McCabe
Remove updatePosition argument from DFSInputStream#getBlockAt()
- HDFS-7494.
Minor bug reported by Ted Yu and fixed by Ted Yu
Checking of closed in DFSInputStream#pread() should be protected by synchronization
- HDFS-7491.
Minor improvement reported by Ming Ma and fixed by Ming Ma (datanode)
Add incremental blockreport latency to DN metrics
- HDFS-7490.
Major bug reported by Steve Loughran and fixed by Steve Loughran (build , test)
HDFS tests OOM on Java7+
- HDFS-7486.
Major sub-task reported by Haohui Mai and fixed by Haohui Mai
Consolidate XAttr-related implementation into a single class
- HDFS-7484.
Major improvement reported by Haohui Mai and fixed by Jing Zhao
Make FSDirectory#addINode take existing INodes as its parameter
- HDFS-7481.
Minor bug reported by Vinayakumar B and fixed by Vinayakumar B (namenode)
Add ACL indicator to the "Permission Denied" exception.
- HDFS-7478.
Major improvement reported by Li Lu and fixed by Li Lu
Move org.apache.hadoop.hdfs.server.namenode.NNConf to FSNamesystem
- HDFS-7476.
Major sub-task reported by Haohui Mai and fixed by Haohui Mai (namenode)
Consolidate ACL-related operations to a single class
- HDFS-7475.
Major test reported by Xiaoyu Yao and fixed by Xiaoyu Yao (test)
Make TestLazyPersistFiles#testLazyPersistBlocksAreSaved deterministic
- HDFS-7474.
Major sub-task reported by Jing Zhao and fixed by Jing Zhao (namenode)
Avoid resolving path in FSPermissionChecker
- HDFS-7473.
Major bug reported by Jason Keller and fixed by Akira AJISAKA (documentation)
Document setting dfs.namenode.fs-limits.max-directory-items to 0 is invalid
- HDFS-7472.
Trivial bug reported by Masatake Iwasaki and fixed by Masatake Iwasaki
Fix typo in message of ReplicaNotFoundException
- HDFS-7470.
Major bug reported by zhaoyunjiong and fixed by zhaoyunjiong (namenode)
SecondaryNameNode need twice memory when calling reloadFromImageFile
- HDFS-7468.
Major sub-task reported by Li Lu and fixed by Li Lu
Moving verify* functions to corresponding classes
- HDFS-7467.
Major sub-task reported by Benoy Antony and fixed by Benoy Antony (balancer & mover)
Provide storage tier information for a directory via fsck
- HDFS-7463.
Major improvement reported by Haohui Mai and fixed by Haohui Mai
Simplify FSNamesystem#getBlockLocationsUpdateTimes
- HDFS-7462.
Major sub-task reported by Haohui Mai and fixed by Haohui Mai
Consolidate implementation of mkdirs() into a single class
- HDFS-7459.
Major sub-task reported by Haohui Mai and fixed by Haohui Mai
Consolidate cache-related implementation in FSNamesystem into a single class
- HDFS-7458.
Minor improvement reported by Yongjun Zhang and fixed by Yongjun Zhang (nfs , test)
Add description to the nfs ports in core-site.xml used by nfs test to avoid confusion
- HDFS-7457.
Major bug reported by Daryn Sharp and fixed by Daryn Sharp (namenode)
DatanodeID generates excessive garbage
Thanks for the reviews, gentlemen. I've committed this to trunk and branch-2. Thanks for identifying and working on the issue, Daryn.
- HDFS-7456.
Major bug reported by Vinayakumar B and fixed by Vinayakumar B (namenode)
De-duplicate AclFeature instances with same AclEntries do reduce memory footprint of NameNode
- HDFS-7454.
Major improvement reported by Vinayakumar B and fixed by Vinayakumar B (namenode)
Reduce memory footprint for AclEntries in NameNode
- HDFS-7450.
Major sub-task reported by Haohui Mai and fixed by Haohui Mai
Consolidate the implementation of GetFileInfo(), GetListings() and GetContentSummary() into a single class
- HDFS-7449.
Major new feature reported by Brandon Li and fixed by Brandon Li (nfs)
Add metrics to NFS gateway
- HDFS-7448.
Minor test reported by Ted Yu and fixed by Akira AJISAKA
TestBookKeeperHACheckpoints fails in trunk build
- HDFS-7446.
Major improvement reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (hdfs-client)
HDFS inotify should have the ability to determine what txid it has read up to
- HDFS-7444.
Major bug reported by Haohui Mai and fixed by Haohui Mai
convertToBlockUnderConstruction should preserve BlockCollection
- HDFS-7440.
Major sub-task reported by Haohui Mai and fixed by Haohui Mai
Consolidate snapshot related operations in a single class
- HDFS-7439.
Minor improvement reported by Ming Ma and fixed by Takanobu Asanuma (balancer & mover , datanode , hdfs-client)
Add BlockOpResponseProto's message to DFSClient's exception message
- HDFS-7438.
Major sub-task reported by Haohui Mai and fixed by Haohui Mai
Consolidate the implementation of rename() into a single class
- HDFS-7435.
Critical improvement reported by Daryn Sharp and fixed by Daryn Sharp (datanode , namenode)
PB encoding of block reports is very inefficient
- HDFS-7434.
Major improvement reported by Daryn Sharp and fixed by Daryn Sharp (namenode)
DatanodeID hashCode should not be mutable
- HDFS-7431.
Major bug reported by Yi Liu and fixed by Yi Liu (security)
log message for InvalidMagicNumberException may be incorrect
- HDFS-7430.
Major improvement reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe
Rewrite the BlockScanner to use O(1) memory and use multiple threads
- HDFS-7426.
Major improvement reported by Andrew Wang and fixed by Andrew Wang (namenode)
Change nntop JMX format to be a JSON blob
- HDFS-7424.
Major new feature reported by Brandon Li and fixed by Brandon Li (nfs)
Add web UI for NFS gateway
- HDFS-7423.
Trivial bug reported by Charles Lamb and fixed by Charles Lamb (nfs)
various typos and message formatting fixes in nfs daemon and doc
- HDFS-7420.
Major sub-task reported by Haohui Mai and fixed by Haohui Mai
Delegate permission checks to FSDirectory
- HDFS-7419.
Major improvement reported by Lei (Eddy) Xu and fixed by Lei (Eddy) Xu (datanode)
Improve error messages for DataNode hot swap drive feature
- HDFS-7415.
Major sub-task reported by Haohui Mai and fixed by Haohui Mai
Move FSNameSystem.resolvePath() to FSDirectory
- HDFS-7413.
Major sub-task reported by Haohui Mai and fixed by Haohui Mai (test)
Some unit tests should use NameNodeProtocols instead of FSNameSystem
- HDFS-7412.
Major sub-task reported by Haohui Mai and fixed by Haohui Mai
Move RetryCache to NameNodeRpcServer
- HDFS-7411.
Major improvement reported by Andrew Wang and fixed by Andrew Wang
Refactor and improve decommissioning logic into DecommissionManager
This change introduces a new configuration key used to throttle decommissioning work, "dfs.namenode.decommission.blocks.per.interval". This new key overrides and deprecates the previous related configuration key "dfs.namenode.decommission.nodes.per.interval". The new key is intended to result in more predictable pause times while scanning decommissioning nodes.
- HDFS-7410.
Major improvement reported by Vinayakumar B and fixed by Vinayakumar B (hdfs-client)
Support CreateFlags with append() to support hsync() for appending streams
- HDFS-7409.
Minor improvement reported by Andrew Wang and fixed by Andrew Wang
Allow dead nodes to finish decommissioning if all files are fully replicated
- HDFS-7406.
Major bug reported by Haohui Mai and fixed by Haohui Mai
SimpleHttpProxyHandler puts incorrect "Connection: Close" header
- HDFS-7404.
Major improvement reported by Haohui Mai and fixed by Li Lu
Remove o.a.h.hdfs.server.datanode.web.resources
- HDFS-7403.
Trivial bug reported by Yongjun Zhang and fixed by Yongjun Zhang (namenode)
Inaccurate javadoc of BlockUCState#COMPLETE state
- HDFS-7399.
Minor bug reported by Ted Yu and fixed by Vinayakumar B
Lack of synchronization in DFSOutputStream#Packet#getLastByteOffsetBlock()
- HDFS-7398.
Major improvement reported by Gera Shegalov and fixed by Gera Shegalov (namenode)
Reset cached thread-local FSEditLogOp's on every FSEditLog#logEdit
- HDFS-7395.
Major bug reported by Yongjun Zhang and fixed by Haohui Mai (namenode)
BlockIdManager#clear() bails out when resetting the GenerationStampV1Limit
- HDFS-7394.
Minor bug reported by Kihwal Lee and fixed by Keith Pak
Log at INFO level, not WARN level, when InvalidToken is seen in ShortCircuitCache
- HDFS-7389.
Major bug reported by Chunjun Xiao and fixed by Vinayakumar B (namenode)
Named user ACL cannot stop the user from accessing the FS entity.
- HDFS-7386.
Trivial improvement reported by Yongjun Zhang and fixed by Yongjun Zhang (datanode , security)
Replace check "port number < 1024" with shared isPrivilegedPort method
- HDFS-7384.
Major improvement reported by Vinayakumar B and fixed by Vinayakumar B (namenode)
'getfacl' command and 'getAclStatus' output should be in sync
- HDFS-7381.
Major improvement reported by Haohui Mai and fixed by Haohui Mai
Decouple the management of block id and gen stamps from FSNamesystem
- HDFS-7375.
Major improvement reported by Haohui Mai and fixed by Haohui Mai
Move FSClusterStats to o.a.h.h.hdfs.server.blockmanagement
- HDFS-7374.
Major bug reported by Zhe Zhang and fixed by Zhe Zhang
Allow decommissioning of dead DataNodes
- HDFS-7373.
Major bug reported by Kihwal Lee and fixed by Kihwal Lee
Clean up temporary files after fsimage transfer failures
- HDFS-7366.
Minor bug reported by Haohui Mai and fixed by Li Lu
BlockInfo should take replication as an short in the constructor
- HDFS-7365.
Minor improvement reported by Li Lu and fixed by Li Lu
Remove hdfs.server.blockmanagement.MutableBlockCollection
- HDFS-7361.
Minor bug reported by Chris Nauroth and fixed by Konstantin Shvachko (datanode , namenode , test)
TestCheckpoint#testStorageAlreadyLockedErrorMessage fails after change of log message related to locking violation.
- HDFS-7358.
Major bug reported by Tsz Wo Nicholas Sze and fixed by Tsz Wo Nicholas Sze (hdfs-client)
Clients may get stuck waiting when using ByteArrayManager
- HDFS-7357.
Minor improvement reported by Tsz Wo Nicholas Sze and fixed by Tsz Wo Nicholas Sze (namenode)
FSNamesystem.checkFileProgress should log file path
- HDFS-7356.
Minor improvement reported by Haohui Mai and fixed by Li Lu (nfs)
Use DirectoryListing.hasMore() directly in nfs
- HDFS-7336.
Major improvement reported by Konstantin Shvachko and fixed by Milan Desai (hdfs-client)
Unused member DFSInputStream.buffersize
- HDFS-7335.
Major improvement reported by Konstantin Shvachko and fixed by Milan Desai (namenode)
Redundant checkOperation() in FSN.analyzeFileState()
- HDFS-7333.
Major improvement reported by Konstantin Shvachko and fixed by Konstantin Shvachko (datanode , namenode)
Improve log message in Storage.tryLock()
- HDFS-7331.
Minor improvement reported by Charles Lamb and fixed by Charles Lamb (datanode)
Add Datanode network counts to datanode jmx page
- HDFS-7329.
Major improvement reported by Konstantin Shvachko and fixed by Byron Wong (test)
MiniDFSCluster should log the exception when createNameNodesAndSetConf() fails.
- HDFS-7326.
Minor improvement reported by Colin Patrick McCabe and fixed by Vijay Bhat (documentation)
Add documentation for hdfs debug commands
Added documentation for the hdfs debug commands to the following URL in the documentation website.
hadoop-project-dist/hadoop-hdfs/HDFSCommands.html
In order to view the new documentation, build the website in a staging area:
$ mvn clean site; mvn site:stage -DstagingDirectory=/tmp/hadoop-site
Point your browser to
file:///tmp/hadoop-site/hadoop-project/hadoop-project-dist/hadoop-hdfs/HDFSCommands.html
- HDFS-7323.
Major improvement reported by Tsz Wo Nicholas Sze and fixed by Jing Zhao (hdfs-client)
Move the get/setStoragePolicy commands out from dfsadmin
- HDFS-7315.
Trivial bug reported by Plamen Jeliazkov and fixed by Plamen Jeliazkov
DFSTestUtil.readFileBuffer opens extra FSDataInputStream
- HDFS-7310.
Major improvement reported by Uma Maheswara Rao G and fixed by Vinayakumar B (balancer & mover)
Mover can give first priority to local DN if it has target storage type available in local DN
- HDFS-7308.
Minor improvement reported by Tsz Wo Nicholas Sze and fixed by Takuya Fukudome (hdfs-client)
DFSClient write packet size may > 64kB
- HDFS-7303.
Minor bug reported by Benoy Antony and fixed by Benoy Antony
NN UI fails to distinguish datanodes on the same host
- HDFS-7301.
Minor bug reported by Zhe Zhang and fixed by Zhe Zhang
TestMissingBlocksAlert should use MXBeans instead of old web UI
- HDFS-7283.
Trivial improvement reported by Stephen Chu and fixed by Stephen Chu (datanode)
Bump DataNode OOM log from WARN to ERROR
- HDFS-7282.
Major bug reported by Jinghui Wang and fixed by Jinghui Wang (test)
Fix intermittent TestShortCircuitCache and TestBlockReaderFactory failures resulting from TemporarySocketDirectory GC
- HDFS-7280.
Major improvement reported by Haohui Mai and fixed by Haohui Mai
Use netty 4 in WebImageViewer
- HDFS-7279.
Major improvement reported by Haohui Mai and fixed by Haohui Mai (datanode , webhdfs)
Use netty to implement DatanodeWebHdfsMethods
- HDFS-7278.
Major improvement reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (datanode)
Add a command that allows sysadmins to manually trigger full block reports from a DN
- HDFS-7277.
Minor bug reported by Haohui Mai and fixed by Haohui Mai (build)
Remove explicit dependency on netty 3.2 in BKJournal
- HDFS-7270.
Major improvement reported by Haohui Mai and fixed by Haohui Mai (datanode)
Add congestion signaling capability to DataNode write protocol
- HDFS-7266.
Minor improvement reported by Gopal V and fixed by Andrew Wang (hdfs-client)
HDFS Peercache enabled check should not lock on object
- HDFS-7263.
Major bug reported by Konstantin Shvachko and fixed by Tao Luo (hdfs-client)
Snapshot read can reveal future bytes for appended files.
- HDFS-7258.
Minor bug reported by Xiaoyu Yao and fixed by Xiaoyu Yao (namenode)
CacheReplicationMonitor rescan schedule log should use DEBUG level instead of INFO level
- HDFS-7257.
Minor improvement reported by Charles Lamb and fixed by Charles Lamb (namenode)
Add the time of last HA state transition to NN's /jmx page
- HDFS-7254.
Major sub-task reported by Lei (Eddy) Xu and fixed by Lei (Eddy) Xu (datanode)
Add documentation for hot swaping DataNode drives
- HDFS-7252.
Trivial improvement reported by Yi Liu and fixed by Yi Liu
small refinement to the use of isInAnEZ in FSNamesystem
- HDFS-7242.
Minor improvement reported by Yi Liu and fixed by Yi Liu (namenode)
Code improvement for FSN#checkUnreadableBySuperuser
- HDFS-7235.
Major bug reported by Yongjun Zhang and fixed by Yongjun Zhang (datanode , namenode)
DataNode#transferBlock should report blocks that don't exist using reportBadBlock
- HDFS-7232.
Trivial bug reported by Zoran Dimitrijevic and fixed by Zoran Dimitrijevic
Populate hostname in httpfs audit log
- HDFS-7227.
Minor bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe
Fix findbugs warning about NP_DEREFERENCE_OF_READLINE_VALUE in SpanReceiverHost
- HDFS-7225.
Major bug reported by Zhe Zhang and fixed by Zhe Zhang (namenode)
Remove stale block invalidation work when DN re-registers with different UUID
- HDFS-7224.
Major bug reported by Eric Payne and fixed by Eric Payne (webhdfs)
Allow reuse of NN connections via webhdfs
- HDFS-7223.
Minor improvement reported by Masatake Iwasaki and fixed by Masatake Iwasaki
Tracing span description of IPC client is too long
- HDFS-7222.
Minor new feature reported by Charles Lamb and fixed by Charles Lamb (datanode)
Expose DataNode network errors as a metric
- HDFS-7213.
Critical bug reported by Daryn Sharp and fixed by Eric Payne (namenode)
processIncrementalBlockReport performance degradation
- HDFS-7210.
Major improvement reported by Vinayakumar B and fixed by Vinayakumar B (hdfs-client , namenode)
Avoid two separate RPC's namenode.append() and namenode.getFileInfo() for an append call from DFSClient
- HDFS-7209.
Major sub-task reported by Yi Liu and fixed by Yi Liu (encryption , performance)
Populate EDEK cache when creating encryption zone
- HDFS-7202.
Minor bug reported by Masatake Iwasaki and fixed by Masatake Iwasaki
Should be able to omit package name of SpanReceiver on "hadoop trace -add"
- HDFS-7201.
Major bug reported by Konstantin Shvachko and fixed by Dawson Choong
Fix typos in hdfs-default.xml
- HDFS-7198.
Trivial bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe
Fix findbugs "unchecked conversion" warning in DFSClient#getPathTraceScope
- HDFS-7194.
Trivial bug reported by Yongjun Zhang and fixed by Yongjun Zhang
Fix findbugs "inefficient new String constructor" warning in DFSClient#PATH
- HDFS-7190.
Major improvement reported by Konstantin Shvachko and fixed by Dawson Choong (namenode)
Bad use of Preconditions in startFileInternal()
- HDFS-7189.
Major sub-task reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (datanode , namenode)
Add trace spans for DFSClient metadata operations
- HDFS-7186.
Minor improvement reported by Masatake Iwasaki and fixed by Masatake Iwasaki (documentation)
Document the "hadoop trace" command.
- HDFS-7182.
Major improvement reported by Ming Ma and fixed by Ming Ma
JMX metrics aren't accessible when NN is busy
- HDFS-7165.
Major improvement reported by Andrew Wang and fixed by Zhe Zhang (namenode)
Separate block metrics for files with replication count 1
- HDFS-7146.
Major bug reported by Yongjun Zhang and fixed by Yongjun Zhang (nfs)
NFS ID/Group lookup requires SSSD enumeration on the server
- HDFS-7097.
Critical bug reported by Kihwal Lee and fixed by Kihwal Lee
Allow block reports to be processed during checkpointing on standby name node
- HDFS-7058.
Major sub-task reported by Konstantin Shvachko and fixed by Dasha Boudnik (test)
Tests for truncate CLI
- HDFS-7056.
Major sub-task reported by Konstantin Shvachko and fixed by Plamen Jeliazkov (namenode)
Snapshot support for truncate
- HDFS-7055.
Major sub-task reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (datanode , namenode)
Add tracing to DFSInputStream
- HDFS-7054.
Major sub-task reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (datanode , namenode)
Make DFSOutputStream tracing more fine-grained
- HDFS-7035.
Major sub-task reported by Lei (Eddy) Xu and fixed by Lei (Eddy) Xu (datanode)
Make adding a new data directory to the DataNode an atomic operation and improve error handling
- HDFS-7026.
Trivial improvement reported by Yongjun Zhang and fixed by Yongjun Zhang
Introduce a string constant for "Failed to obtain user group info..."
- HDFS-7009.
Major bug reported by Ming Ma and fixed by Ming Ma (datanode)
Active NN and standby NN have different live nodes
- HDFS-7008.
Minor bug reported by Ted Yu and fixed by Tsuyoshi Ozawa
xlator should be closed upon exit from DFSAdmin#genericRefresh()
- HDFS-6982.
Major new feature reported by Maysam Yabandeh and fixed by Maysam Yabandeh
nntop: top-like tool for name node users
- HDFS-6938.
Trivial bug reported by Charles Lamb and fixed by Charles Lamb (namenode)
Cleanup javac warnings in FSNamesystem
- HDFS-6917.
Major bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (hdfs-client)
Add an hdfs debug command to validate blocks, call recoverlease, etc.
- HDFS-6877.
Major sub-task reported by Lei (Eddy) Xu and fixed by Lei (Eddy) Xu (datanode)
Avoid calling checkDisk when an HDFS volume is removed during a write.
- HDFS-6841.
Major bug reported by Vinayakumar B and fixed by Vinayakumar B
Use Time.monotonicNow() wherever applicable instead of Time.now()
- HDFS-6833.
Critical bug reported by Shinichi Yamashita and fixed by Shinichi Yamashita (datanode)
DirectoryScanner should not register a deleting block with memory of DataNode
- HDFS-6826.
Major new feature reported by Alejandro Abdelnur and fixed by Arun Suresh (security)
Plugin interface to enable delegation of HDFS authorization assertions
- HDFS-6824.
Minor sub-task reported by Andrew Wang and fixed by Andrew Wang (documentation)
Additional user documentation for HDFS encryption.
- HDFS-6806.
Minor improvement reported by Akira AJISAKA and fixed by J.Andreina (documentation)
HDFS Rolling upgrade document should mention the versions available
- HDFS-6803.
Major sub-task reported by stack and fixed by stack (hdfs-client)
Documenting DFSClient#DFSInputStream expectations reading and preading in concurrent context
- HDFS-6753.
Major bug reported by J.Andreina and fixed by J.Andreina
Initialize checkDisk when DirectoryScanner not able to get files list for scanning
- HDFS-6741.
Trivial improvement reported by Stephen Chu and fixed by Harsh J
Improve permission denied message when FSPermissionChecker#checkOwner fails
- HDFS-6735.
Major improvement reported by Liang Xie and fixed by Lars Hofhansl (hdfs-client)
A minor optimization to avoid pread() be blocked by read() inside the same DFSInputStream
- HDFS-6673.
Minor sub-task reported by Lei (Eddy) Xu and fixed by Lei (Eddy) Xu
Add delimited format support to PB OIV tool
- HDFS-6663.
Major new feature reported by Kihwal Lee and fixed by Chen He
Admin command to track file and locations from block id
- HDFS-6662.
Critical bug reported by Brahma Reddy Battula and fixed by Gerson Carlos (namenode)
WebHDFS cannot open a file if its path contains "%"
- HDFS-6657.
Minor bug reported by Vinayakumar B and fixed by Vinayakumar B
Remove link to 'Legacy UI' in Namenode UI
- HDFS-6651.
Critical bug reported by Kihwal Lee and fixed by Jing Zhao
Deletion failure can leak inodes permanently
- HDFS-6565.
Major improvement reported by Haohui Mai and fixed by Akira AJISAKA
Use jackson instead jetty json in hdfs-client
- HDFS-6538.
Trivial bug reported by debugging and fixed by David Luo (datanode)
Comment format error in ShortCircuitRegistry javadoc
- HDFS-6488.
Major new feature reported by Stephen Chu and fixed by Brandon Li (nfs)
Support HDFS superuser in NFSv3 gateway
- HDFS-6425.
Major bug reported by Ming Ma and fixed by Ming Ma
Large postponedMisreplicatedBlocks has impact on blockReport latency
- HDFS-6252.
Minor improvement reported by Fengdong Yu and fixed by Haohui Mai (namenode)
Phase out the old web UI in HDFS
- HDFS-6133.
Major improvement reported by zhaoyunjiong and fixed by zhaoyunjiong (balancer & mover , datanode)
Make Balancer support exclude specified path
Add a feature for replica pinning so that when a replica is pinned in a datanode, it will not be moved by Balancer/Mover. The replica pinning feature can be enabled/disabled by "dfs.datanode.block-pinning.enabled", where the default is false.
- HDFS-5928.
Major sub-task reported by Siqi Li and fixed by Siqi Li
show namespace and namenode ID on NN dfshealth page
- HDFS-5853.
Minor improvement reported by Akira AJISAKA and fixed by Akira AJISAKA (documentation , namenode)
Add "hadoop.user.group.metrics.percentiles.intervals" to hdfs-default.xml
- HDFS-5782.
Minor sub-task reported by David Powell and fixed by Joe Pallas (datanode)
BlockListAsLongs should take lists of Replicas rather than concrete classes
- HDFS-5631.
Minor sub-task reported by David Powell and fixed by Joe Pallas (datanode)
Expose interfaces required by FsDatasetSpi implementations
- HDFS-5578.
Minor bug reported by Andrew Purtell and fixed by Andrew Purtell
[JDK8] Fix Javadoc errors caused by incorrect or illegal tags in doc comments
- HDFS-5445.
Minor bug reported by Jonathan Mace and fixed by Jonathan Mace (datanode)
PacketReceiver populates the packetLen field in PacketHeader incorrectly
- HDFS-4266.
Major sub-task reported by Ivan Kelly and fixed by Rakesh R (ha)
BKJM: Separate write and ack quorum
- HDFS-4265.
Major sub-task reported by Ivan Kelly and fixed by Rakesh R (ha)
BKJM doesn't take advantage of speculative reads
- HDFS-3689.
Major new feature reported by Suresh Srinivas and fixed by Jing Zhao (datanode , hdfs-client , namenode)
Add support for variable length block
1. HDFS now can choose to append data to a new block instead of end of the last partial block. Users can pass {{CreateFlag.APPEND}} and {{CreateFlag.NEW_BLOCK}} to the {{append}} API to indicate this requirement.
2. HDFS now allows users to pass {{SyncFlag.END_BLOCK}} to the {{hsync}} API to finish the current block and write remaining data to a new block.
- HDFS-3519.
Critical bug reported by Todd Lipcon and fixed by Ming Ma (namenode)
Checkpoint upload may interfere with a concurrent saveNamespace
- HDFS-3342.
Minor improvement reported by Todd Lipcon and fixed by Yongjun Zhang (datanode)
SocketTimeoutException in BlockSender.sendChunks could have a better error message
- HDFS-3107.
Major new feature reported by Lei Chang and fixed by Plamen Jeliazkov (datanode , namenode)
HDFS truncate
- HDFS-2605.
Major bug reported by Konstantin Shvachko and fixed by Allen Wittenauer (documentation)
CHANGES.txt has two "Release 0.21.1" sections
- HDFS-2486.
Minor task reported by Steve Loughran and fixed by Uma Maheswara Rao G (namenode)
Review issues with UnderReplicatedBlocks
- HDFS-2219.
Minor improvement reported by Jitendra Nath Pandey and fixed by Tsz Wo Nicholas Sze (tools)
Fsck should work with fully qualified file paths.
- HDFS-1522.
Major bug reported by Konstantin Shvachko and fixed by Dongming Liang (datanode)
Merge Block.BLOCK_FILE_PREFIX and DataStorage.BLOCK_FILE_PREFIX into one constant
This merges Block.BLOCK_FILE_PREFIX and DataStorage.BLOCK_FILE_PREFIX into one constant. Hard-coded
literals of "blk_" in various files are also updated to use the same constant.
- HDFS-1362.
Major new feature reported by Wang Xu and fixed by Wang Xu (datanode)
Provide volume management functionality for DataNode
Based on the reconfiguration framework provided by HADOOP-7001, enable reconfigure the dfs.datanode.data.dir and add new volumes into service.
- HDFS-316.
Minor improvement reported by Brian Bockelman and fixed by Xiaoyu Yao (balancer & mover)
Balancer should run for a configurable # of iterations
- HDFS-49.
Minor bug reported by Steve Loughran and fixed by Steve Loughran (test)
MiniDFSCluster.stopDataNode will always shut down a node in the cluster if a matching name is not found
- HADOOP-11815.
Blocker bug reported by Rohith and fixed by Rohith
HttpServer2 should destroy SignerSecretProvider when it stops
- HADOOP-11801.
Minor improvement reported by Gabor Liptak and fixed by Gabor Liptak (documentation)
Update BUILDING.txt for Ubuntu
ProtocolBuffer is packaged in Ubuntu
- HADOOP-11796.
Minor sub-task reported by Xiaoyu Yao and fixed by Xiaoyu Yao (test)
Skip TestShellBasedIdMapping.testStaticMapUpdate on Windows
- HADOOP-11787.
Critical bug reported by Colin Patrick McCabe and fixed by Kiran Kumar M R (native)
OpensslSecureRandom.c pthread_threadid_np usage signature is wrong on 32-bit Mac
- HADOOP-11776.
Blocker bug reported by Li Lu and fixed by Li Lu
jdiff is broken in Hadoop 2
- HADOOP-11761.
Minor bug reported by Li Lu and fixed by Li Lu
Fix findbugs warnings in org.apache.hadoop.security.authentication
- HADOOP-11757.
Major bug reported by Brandon Li and fixed by Brandon Li (nfs)
NFS gateway should shutdown when it can't start UDP or TCP server
- HADOOP-11754.
Blocker bug reported by Sangjin Lee and fixed by Haohui Mai
RM fails to start in non-secure mode due to authentication filter failure
- HADOOP-11748.
Critical bug reported by Haohui Mai and fixed by Li Lu
The secrets of auth cookies should not be specified in configuration in clear text
- HADOOP-11738.
Major bug reported by Tsuyoshi Ozawa and fixed by Tsuyoshi Ozawa
Fix a link of Protocol Buffers 2.5 for download in BUILDING.txt
- HADOOP-11729.
Minor bug reported by Masatake Iwasaki and fixed by Masatake Iwasaki (documentation)
Fix link to cgroups doc in site.xml
Committed this to trunk, branch-2, and branch-2.7. Thanks Masatake for your contribution!
- HADOOP-11722.
Major bug reported by Arun Suresh and fixed by Arun Suresh
Some Instances of Services using ZKDelegationTokenSecretManager go down when old token cannot be deleted
- HADOOP-11720.
Major bug reported by Akira AJISAKA and fixed by Akira AJISAKA (documentation)
[JDK8] Fix javadoc errors caused by incorrect or illegal tags in hadoop-tools
- HADOOP-11714.
Minor improvement reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (tracing)
Add more trace log4j messages to SpanReceiverHost
- HADOOP-11710.
Critical sub-task reported by Sean Busbey and fixed by Sean Busbey (fs)
Make CryptoOutputStream behave like DFSOutputStream wrt synchronization
- HADOOP-11693.
Major bug reported by Duo Xu and fixed by Duo Xu (tools)
Azure Storage FileSystem rename operations are throttled too aggressively to complete HBase WAL archiving.
- HADOOP-11691.
Critical bug reported by Remus Rusanu and fixed by Kiran Kumar M R (build , native)
X86 build of libwinutils is broken
- HADOOP-11686.
Major bug reported by zhangduo and fixed by zhangduo (security , test)
MiniKDC cannot change ORG_NAME or ORG_DOMAIN
- HADOOP-11674.
Critical bug reported by Sean Busbey and fixed by Sean Busbey (io)
oneByteBuf in CryptoInputStream and CryptoOutputStream should be non static
- HADOOP-11670.
Blocker sub-task reported by Adam Budde and fixed by Adam Budde (fs/s3)
Regression: s3a auth setup broken
- HADOOP-11666.
Major bug reported by Akira AJISAKA and fixed by Byron Wong
Revert the format change of du output introduced by HADOOP-6857
- HADOOP-11658.
Minor improvement reported by Kai Zheng and fixed by Kai Zheng
Externalize io.compression.codecs property
- HADOOP-11648.
Major improvement reported by Liang Xie and fixed by Liang Xie (net)
Set DomainSocketWatcher thread name explicitly
- HADOOP-11642.
Major improvement reported by shashank and fixed by shashank (tools)
Upgrade azure sdk version from 0.6.0 to 2.0.0
- HADOOP-11639.
Major bug reported by Chris Nauroth and fixed by Remus Rusanu (native)
Clean up Windows native code compilation warnings related to Windows Secure Container Executor.
- HADOOP-11638.
Major bug reported by Dmitry Sivachenko and fixed by Kiran Kumar M R (native)
OpensslSecureRandom.c pthreads_thread_id should support FreeBSD and Solaris in addition to Linux
- HADOOP-11634.
Major bug reported by Brahma Reddy Battula and fixed by Brahma Reddy Battula (documentation)
Description of webhdfs' principal/keytab should switch places each other
- HADOOP-11633.
Major improvement reported by Colin Patrick McCabe and fixed by Masatake Iwasaki (documentation)
Convert remaining branch-2 .apt.vm files to markdown
- HADOOP-11632.
Minor improvement reported by Akira AJISAKA and fixed by Akira AJISAKA
Cleanup Find.java to remove SupressWarnings annotations
- HADOOP-11629.
Major bug reported by shanyu zhao and fixed by shanyu zhao (tools)
WASB filesystem should not start BandwidthGaugeUpdater if fs.azure.skip.metrics set to true
- HADOOP-11620.
Major improvement reported by Arun Suresh and fixed by Arun Suresh (kms)
Add support for load balancing across a group of KMS for HA
- HADOOP-11619.
Major bug reported by Gera Shegalov and fixed by Brahma Reddy Battula (fs)
FTPFileSystem should override getDefaultPort
- HADOOP-11618.
Major bug reported by Gera Shegalov and fixed by Brahma Reddy Battula (fs)
DelegateToFileSystem erroneously uses default FS's port in constructor
- HADOOP-11615.
Minor bug reported by Akira AJISAKA and fixed by Brahma Reddy Battula (documentation)
Update ServiceLevelAuth.md for YARN
- HADOOP-11612.
Major task reported by Robert Kanter and fixed by Robert Kanter
Workaround for Curator's ChildReaper requiring Guava 15+
- HADOOP-11609.
Major bug reported by Brahma Reddy Battula and fixed by Varun Saxena (documentation , security)
Correct credential commands info in CommandsManual.html#credential
- HADOOP-11607.
Trivial improvement reported by Lei (Eddy) Xu and fixed by Lei (Eddy) Xu (fs/s3)
Reduce log spew in S3AFileSystem
- HADOOP-11605.
Minor bug reported by Gera Shegalov and fixed by Gera Shegalov (fs)
FilterFileSystem#create with ChecksumOpt should propagate it to wrapped FS
- HADOOP-11604.
Critical bug reported by Liang Xie and fixed by Chris Nauroth (net)
Prevent ConcurrentModificationException while closing domain sockets during shutdown of DomainSocketWatcher thread.
- HADOOP-11602.
Blocker bug reported by Tsuyoshi Ozawa and fixed by Tsuyoshi Ozawa
Fix toUpperCase/toLowerCase to use Locale.ENGLISH
- HADOOP-11600.
Major sub-task reported by Tsuyoshi Ozawa and fixed by Tsuyoshi Ozawa
Fix up source codes to be compiled with Guava 17.0
- HADOOP-11599.
Minor improvement reported by zhihai xu and fixed by zhihai xu (ipc)
Client#getTimeout should use IPC_CLIENT_PING_DEFAULT when IPC_CLIENT_PING_KEY is not configured.
- HADOOP-11595.
Major bug reported by Yi Liu and fixed by Yi Liu (fs)
Add default implementation for AbstractFileSystem#truncate
- HADOOP-11589.
Minor improvement reported by Akira AJISAKA and fixed by Rakesh R (net)
NetUtils.createSocketAddr should trim the input URI
- HADOOP-11587.
Trivial bug reported by Xiaoyu Yao and fixed by Xiaoyu Yao (test)
TestMapFile#testMainMethodMapFile creates test files in hadoop-common project root
- HADOOP-11586.
Minor improvement reported by Ray Chiang and fixed by Ray Chiang (metrics)
Update use of Iterator to Iterable in AbstractMetricsContext.java
- HADOOP-11584.
Blocker sub-task reported by Dan Hecht and fixed by Brahma Reddy Battula (fs/s3)
s3a file block size set to 0 in getFileStatus
- HADOOP-11579.
Major improvement reported by Steve Loughran and fixed by Konstantin Shvachko (documentation)
Documentation for truncate
- HADOOP-11571.
Blocker bug reported by Steve Loughran and fixed by Steve Loughran (fs/s3)
Über-jira: S3a stabilisation phase I
- HADOOP-11570.
Major sub-task reported by Dan Hecht and fixed by Dan Hecht (fs/s3)
S3AInputStream.close() downloads the remaining bytes of the object from S3
- HADOOP-11569.
Major improvement reported by Vinayakumar B and fixed by Vinayakumar B
Provide Merge API for MapFile to merge multiple similar MapFiles to one MapFile
- HADOOP-11558.
Minor bug reported by Masatake Iwasaki and fixed by Jean-Pierre Matsumoto (documentation)
Fix dead links to doc of hadoop-tools
- HADOOP-11549.
Major bug reported by Yongjun Zhang and fixed by Yongjun Zhang (tools)
flaky test detection tool failed to handle special control characters in test result
- HADOOP-11548.
Major bug reported by Colin Patrick McCabe and fixed by Anu Engineer (build , native)
checknative should display a nicer error message when openssl support is not compiled in
- HADOOP-11547.
Major bug reported by Chris Nauroth and fixed by Chris Nauroth (native)
hadoop-common native compilation fails on Windows due to missing support for __attribute__ declaration.
- HADOOP-11546.
Major bug reported by Steve Loughran and fixed by Tsuyoshi Ozawa (build)
Checkstyle failing: Unable to instantiate DoubleCheckedLockingCheck
- HADOOP-11545.
Minor bug reported by Brahma Reddy Battula and fixed by Brahma Reddy Battula (security)
ArrayIndexOutOfBoundsException is thrown with "hadoop credential list -provider"
- HADOOP-11544.
Trivial improvement reported by Masatake Iwasaki and fixed by Masatake Iwasaki
Remove unused configuration keys for tracing
- HADOOP-11543.
Trivial improvement reported by Jagadesh Kiran N and fixed by Brahma Reddy Battula (scripts)
Improve help message for hadoop/yarn command
- HADOOP-11535.
Minor test reported by Kai Zheng and fixed by Kai Zheng
TableMapping related tests failed due to 'successful' resolving of invalid test hostname
- HADOOP-11529.
Minor bug reported by Masatake Iwasaki and fixed by Masatake Iwasaki (tools)
Fix findbugs warnings in hadoop-archives
- HADOOP-11526.
Major bug reported by Ian Rogers and fixed by Anu Engineer (io , native)
Memory leak in Bzip2Compressor and Bzip2Decompressor
- HADOOP-11523.
Major bug reported by Duo Xu and fixed by Duo Xu (tools)
StorageException complaining " no lease ID" when updating FolderLastModifiedTime in WASB
- HADOOP-11522.
Major sub-task reported by Thomas Demoor and fixed by Thomas Demoor (fs/s3)
Update S3A Documentation
- HADOOP-11521.
Minor sub-task reported by Thomas Demoor and fixed by Thomas Demoor (fs/s3)
Make connection timeout configurable in s3a
- HADOOP-11520.
Minor improvement reported by Thomas Demoor and fixed by Thomas Demoor (fs/s3)
Clean incomplete multi-part uploads in S3A tests
- HADOOP-11512.
Minor bug reported by Harsh J and fixed by Ryan P (io)
Use getTrimmedStrings when reading serialization keys
- HADOOP-11510.
Major new feature reported by Yi Liu and fixed by Yi Liu (fs)
Expose truncate API via FileContext
- HADOOP-11509.
Major bug reported by Xuan Gong and fixed by Xuan Gong
change parsing sequence in GenericOptionsParser to parse -D parameters first
- HADOOP-11507.
Minor bug reported by Talat UYARER and fixed by Talat UYARER
Hadoop RPC Authentication problem with different user locale
- HADOOP-11506.
Major improvement reported by Dmitriy V. Ryaboy and fixed by Gera Shegalov (conf)
Configuration variable expansion regex expensive for long values
- HADOOP-11500.
Major bug reported by Ted Yu and fixed by Ted Yu
InputStream is left unclosed in ApplicationClassLoader
- HADOOP-11499.
Minor bug reported by Ted Yu and fixed by Ted Yu
Check of executorThreadsStarted in ValueQueue#submitRefillTask() evades lock acquisition
- HADOOP-11498.
Major improvement reported by Masatake Iwasaki and fixed by Masatake Iwasaki
Bump the version of HTrace to 3.1.0-incubating
- HADOOP-11497.
Major bug reported by Christian Winkler and fixed by Christian Winkler (documentation)
Fix typo in ClusterSetup.html#Hadoop_Startup
Correct startup command for cluster data nodes
- HADOOP-11495.
Major improvement reported by Allen Wittenauer and fixed by Masatake Iwasaki (documentation)
Convert site documentation from apt to markdown
- HADOOP-11494.
Minor bug reported by Ted Yu and fixed by Ted Yu
Lock acquisition on WrappedInputStream#unwrappedRpcBuffer may race with another thread
- HADOOP-11493.
Trivial bug reported by Charles Lamb and fixed by Charles Lamb (kms)
Fix some typos in kms-acls.xml description
- HADOOP-11492.
Major task reported by Karthik Kambatla and fixed by Arun Suresh
Bump up curator version to 2.7.1
Apache Curator version change: Apache Hadoop has updated the version of Apache Curator used from 2.6.0 to 2.7.1. This change should be binary and source compatible for the majority of downstream users. Notable exceptions:
# Binary incompatible change: org.apache.curator.utils.PathUtils.validatePath(String) changed return types. Downstream users of this method will need to recompile.
# Source incompatible change: org.apache.curator.framework.recipes.shared.SharedCountReader added a method to its interface definition. Downstream users with custom implementations of this interface can continue without binary compatibility problems but will need to modify their source code to recompile.
# Source incompatible change: org.apache.curator.framework.recipes.shared.SharedValueReader added a method to its interface definition. Downstream users with custom implementations of this interface can continue without binary compatibility problems but will need to modify their source code to recompile.
Downstream users are reminded that while the Hadoop community will attempt to avoid egregious incompatible dependency changes, there is currently no policy around when Hadoop's exposed dependencies will change across versions (ref https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/Compatibility.html#Java_Classpath).
- HADOOP-11490.
Major new feature reported by Konstantin Shvachko and fixed by Milan Desai (fs)
Expose truncate API via FileSystem and shell command
- HADOOP-11489.
Minor task reported by Ted Yu and fixed by Ted Yu (nfs)
Dropping dependency on io.netty from hadoop-nfs' pom.xml
- HADOOP-11488.
Minor bug reported by Harsh J and fixed by Daisuke Kobayashi (fs/s3)
Difference in default connection timeout for S3A FS
- HADOOP-11483.
Major improvement reported by Colin Patrick McCabe and fixed by Akira AJISAKA
HardLink.java should use the jdk7 createLink method
- HADOOP-11482.
Major bug reported by Arun Suresh and fixed by Arun Suresh
Use correct UGI when KMSClientProvider is called by a proxy user
- HADOOP-11481.
Minor improvement reported by Yi Yao and fixed by Charles Lamb
ClassCastException while using a key created by keytool to create encryption zone.
- HADOOP-11480.
Minor bug reported by Ted Yu and fixed by Ted Yu (documentation)
Typo in hadoop-aws/index.md uses wrong scheme for test.fs.s3.name
- HADOOP-11470.
Major bug reported by Sangjin Lee and fixed by Sangjin Lee
Remove some uses of obsolete guava APIs from the hadoop codebase
- HADOOP-11469.
Minor bug reported by Dian Fu and fixed by Dian Fu (kms)
KMS should skip default.key.acl and whitelist.key.acl when loading key acl
- HADOOP-11467.
Critical bug reported by Robert Kanter and fixed by Yongjun Zhang (security)
KerberosAuthenticator can connect to a non-secure cluster
- HADOOP-11465.
Major sub-task reported by Varun Saxena and fixed by Varun Saxena
Fix findbugs warnings in hadoop-gridmix
- HADOOP-11464.
Major improvement reported by Chris Nauroth and fixed by Chris Nauroth (bin)
Reinstate support for launching Hadoop processes on Windows using Cygwin.
We have reinstated support for launching Hadoop processes on Windows by using Cygwin to run the shell scripts. All processes still must have access to the native components: hadoop.dll and winutils.exe.
- HADOOP-11463.
Major task reported by Ted Yu and fixed by Ted Yu (fs/s3)
Replace method-local TransferManager object with S3AFileSystem#transfers
- HADOOP-11462.
Major bug reported by Ayappan and fixed by Ayappan (test)
TestSocketIOWithTimeout needs change for PowerPC platform
- HADOOP-11459.
Minor bug reported by Vinayakumar B and fixed by Vinayakumar B
Fix recent findbugs in ActiveStandbyElector, NetUtils and ShellBasedIdMapping
- HADOOP-11455.
Minor improvement reported by Charles Lamb and fixed by Charles Lamb (security)
KMS and Credential CLI should request confirmation for deletion by default
- HADOOP-11450.
Minor bug reported by Tsuyoshi Ozawa and fixed by Varun Saxena (tools/distcp)
Cleanup DistCpV1 not to use deprecated methods and fix javadocs
- HADOOP-11449.
Major bug reported by Alec Taylor and fixed by Chris Nauroth (build)
[JDK8] Cannot build on Windows: error: unexpected end tag: </ul>
- HADOOP-11448.
Minor improvement reported by Akira AJISAKA and fixed by Tsuyoshi Ozawa
Fix findbugs warnings in FileBasedIPList
- HADOOP-11446.
Major bug reported by Ted Yu and fixed by Ted Yu (fs/s3)
S3AOutputStream should use shared thread pool to avoid OutOfMemoryError
The following parameters are introduced in this JIRA:
fs.s3a.threads.max: the maximum number of threads to allow in the pool used by TransferManager
fs.s3a.threads.core: the number of threads to keep in the pool used by TransferManager
fs.s3a.threads.keepalivetime: when the number of threads is greater than the core, this is the maximum time that excess idle threads will wait for new tasks before terminating
fs.s3a.max.total.tasks: the maximum number of tasks that the LinkedBlockingQueue can hold
- HADOOP-11445.
Critical bug reported by Ankit Kamboj and fixed by Ankit Kamboj
Bzip2Codec: Data block is skipped when position of newly created stream is equal to start of split
- HADOOP-11442.
Major improvement reported by shashank and fixed by shashank (tools)
hadoop-azure: Create test jar
- HADOOP-11441.
Minor improvement reported by shashank and fixed by shashank (tools)
Hadoop-azure: Change few methods scope to public
- HADOOP-11440.
Minor improvement reported by Akira AJISAKA and fixed by Kengo Seki
Use "test.build.data" instead of "build.test.dir" for testing in ClientBaseWithFixes
- HADOOP-11432.
Major test reported by Liang Xie and fixed by Liang Xie (fs)
Fix SymlinkBaseTest#testCreateLinkUsingPartQualPath2
- HADOOP-11431.
Major bug reported by Hervé Boutemy and fixed by Hervé Boutemy
clean up redundant maven-site-plugin configuration
- HADOOP-11430.
Major improvement reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe
Add GenericTestUtils#disableLog, GenericTestUtils#setLogLevel
- HADOOP-11429.
Major sub-task reported by Varun Saxena and fixed by Varun Saxena
Findbugs warnings in hadoop extras
- HADOOP-11428.
Major bug reported by Arpit Agarwal and fixed by Arpit Agarwal (documentation)
Remove obsolete reference to Cygwin in BUILDING.txt
- HADOOP-11427.
Major improvement reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe
ChunkedArrayList: fix removal via iterator and implement get
- HADOOP-11422.
Minor improvement reported by Yi Liu and fixed by Yi Liu
Check CryptoCodec is AES-CTR for Crypto input/output stream
- HADOOP-11421.
Major improvement reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe
Add IOUtils#listDirectory
- HADOOP-11420.
Trivial bug reported by Hervé Boutemy and fixed by Hervé Boutemy (site)
Use latest maven-site-plugin and replace link to svn with link to git
- HADOOP-11419.
Minor improvement reported by Hervé Boutemy and fixed by Hervé Boutemy (build)
improve hadoop-maven-plugins
- HADOOP-11416.
Major improvement reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe
Move ChunkedArrayList into hadoop-common
- HADOOP-11414.
Minor bug reported by Ted Yu and fixed by Tsuyoshi Ozawa
FileBasedIPList#readLines() can leak file descriptors
- HADOOP-11412.
Trivial bug reported by Hervé Boutemy and fixed by Hervé Boutemy
POMs mention "The Apache Software License" rather than "Apache License"
- HADOOP-11411.
Major bug reported by Jason Dere and fixed by
Hive build failure on hadoop-2.7 due to HADOOP-11356
- HADOOP-11410.
Major improvement reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe
make the rpath of libhadoop.so configurable
- HADOOP-11409.
Major bug reported by Jason Lowe and fixed by Gera Shegalov
FileContext.getFileContext can stack overflow if default fs misconfigured
- HADOOP-11403.
Major bug reported by Malcolm Kavalsky and fixed by Malcolm Kavalsky
Avoid using sys_errlist on Solaris, which lacks support for it
- HADOOP-11402.
Major bug reported by Colin Patrick McCabe and fixed by Varun Saxena
Negative user-to-group cache entries are never cleared for never-again-accessed users
- HADOOP-11400.
Major bug reported by Kamil Gorlo and fixed by Kamil Gorlo (metrics)
GraphiteSink does not reconnect to Graphite after 'broken pipe'
- HADOOP-11399.
Minor improvement reported by Ray Chiang and fixed by Ray Chiang
Java Configuration file and .xml files should be automatically cross-compared
- HADOOP-11396.
Major improvement reported by Chris Nauroth and fixed by Chris Nauroth (documentation)
Provide navigation in the site documentation linking to the Hadoop Compatible File Systems.
- HADOOP-11395.
Major improvement reported by Chris Nauroth and fixed by Chris Nauroth (documentation)
Add site documentation for Azure Storage FileSystem integration.
- HADOOP-11394.
Major bug reported by Chris Nauroth and fixed by Chris Nauroth (documentation)
hadoop-aws documentation missing.
- HADOOP-11390.
Minor improvement reported by Steve Loughran and fixed by Varun Saxena (metrics)
Metrics 2 ganglia provider to include hostname in unresolved address problems
- HADOOP-11389.
Major sub-task reported by Haohui Mai and fixed by Haohui Mai
Clean up byte to string encoding issues in hadoop-common
- HADOOP-11388.
Minor bug reported by Haohui Mai and fixed by Li Lu
Remove deprecated o.a.h.metrics.file.FileContext
- HADOOP-11386.
Major bug reported by Li Lu and fixed by Li Lu
Replace \n by %n in format hadoop-common format strings
- HADOOP-11385.
Critical bug reported by Haohui Mai and fixed by Haohui Mai
Prevent cross site scripting attack on JMXJSONServlet
- HADOOP-11381.
Major sub-task reported by Li Lu and fixed by Li Lu
Fix findbugs warnings in hadoop-distcp, hadoop-aws, hadoop-azure, and hadoop-openstack
- HADOOP-11379.
Major sub-task reported by Li Lu and fixed by Li Lu
Fix new findbugs warnings in hadoop-auth*
- HADOOP-11378.
Major sub-task reported by Li Lu and fixed by Li Lu
Fix new findbugs warnings in hadoop-kms
- HADOOP-11377.
Major sub-task reported by Steve Loughran and fixed by Tsuyoshi Ozawa (build)
jdiff failing on java 7 and java 8, "Null.java" not found
- HADOOP-11372.
Major sub-task reported by Li Lu and fixed by Li Lu
Fix new findbugs warnings in mapreduce-examples
- HADOOP-11370.
Major sub-task reported by Zhijie Shen and fixed by
Fix new findbug warnings hadoop-yarn
- HADOOP-11369.
Major sub-task reported by Li Lu and fixed by Li Lu
Fix new findbugs warnings in hadoop-mapreduce-client, non-core directories
- HADOOP-11368.
Major bug reported by Arun Suresh and fixed by Arun Suresh (kms)
Fix SSLFactory truststore reloader thread leak in KMSClientProvider
- HADOOP-11367.
Major sub-task reported by Li Lu and fixed by Li Lu
Fix warnings from findbugs 3.0 in hadoop-streaming
- HADOOP-11363.
Major bug reported by Steve Loughran and fixed by Steve Loughran (build)
Hadoop maven surefire-plugin uses must set heap size
- HADOOP-11358.
Major test reported by Yi Liu and fixed by Yi Liu (security , test)
Tests for encryption/decryption with IV calculation overflow
- HADOOP-11355.
Minor bug reported by Arun Suresh and fixed by Arun Suresh
When accessing data in HDFS and the key has been deleted, a Null Pointer Exception is shown.
- HADOOP-11354.
Major bug reported by Ted Yu and fixed by Ted Yu
ThrottledInputStream doesn't perform effective throttling
- HADOOP-11350.
Major bug reported by Benoy Antony and fixed by Benoy Antony (security)
The size of header buffer of HttpServer is too small when HTTPS is enabled
- HADOOP-11349.
Minor bug reported by Chris Nauroth and fixed by Varun Saxena (fs)
RawLocalFileSystem leaks file descriptor while creating a file if creat succeeds but chmod fails.
- HADOOP-11348.
Minor bug reported by Dian Fu and fixed by Dian Fu
Remove unused variable from CMake error message for finding openssl
Test failure is unrelated. Committed to 2.7. Thanks, Dian.
- HADOOP-11344.
Major bug reported by Arun Suresh and fixed by Arun Suresh
KMS kms-config.sh sets a default value for the keystore password even in non-ssl setup
- HADOOP-11343.
Blocker bug reported by Jerry Chen and fixed by Jerry Chen (security)
Overflow is not properly handled in caclulating final iv for AES CTR
- HADOOP-11342.
Major bug reported by Dian Fu and fixed by Dian Fu (kms , security)
KMS key ACL should ignore ALL operation for default key ACL and whitelist key ACL
- HADOOP-11341.
Major new feature reported by Arun Suresh and fixed by Arun Suresh (kms , security)
KMS support for whitelist key ACLs
- HADOOP-11337.
Major bug reported by Dian Fu and fixed by Dian Fu
KeyAuthorizationKeyProvider access checks need to be done atomically
- HADOOP-11333.
Major bug reported by zhaoyunjiong and fixed by zhaoyunjiong
Fix deadlock in DomainSocketWatcher when the notification pipe is full
- HADOOP-11332.
Major bug reported by Dian Fu and fixed by Dian Fu (security)
KerberosAuthenticator#doSpnegoSequence should check if kerberos TGT is available in the subject
- HADOOP-11329.