Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 1.979104507s trace: https://jaeger.dev.storj.io/trace/92f373d508d696c FAIL (2025-12-09 09:51:46 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 09:26:29.076085885 (89h57m28.94716892s) [upload=78482]: Starting upload Dec 9 09:26:29.076139197 (89h57m28.947222232s) [upload=78482]: Done starting upload Dec 9 09:26:29.076210933 (89h57m28.947293978s) [upload=78482]: Uploading segments... Dec 9 09:26:29.076295604 (89h57m28.947378639s) [upload=78482]: Got next segment. Inline: true Dec 9 09:26:29.076303269 (89h57m28.947386314s) [upload=78482]: Waiting for error group managing segments... Dec 9 09:26:29.076399502 (89h57m28.947482537s) [upload=78482]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 09:26:29.076427595 (89h57m28.94751064s) [upload=78482]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 9 09:26:29.076432515 (89h57m28.94751555s) [upload=78482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 09:27:28.400202146 (89h58m28.271285192s) [upload=78482]: Done uploading segments... Dec 9 09:27:28.400236482 (89h58m28.271319517s) [upload=78482]: Upload finished. err: metaclient: context deadline exceeded Duration:* 1.979127311s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 4.032516517s remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/754b20513c76ffd8 FAIL (2025-12-09 09:51:37 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 4.032546234s core satellite us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 17.589159377s trace: https://jaeger.dev.storj.io/trace/2278184cf7cb09d3 FAIL (2025-12-09 09:51:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== HDOWUF7N2LWP3MU4DSDXO3YAI6KVCUP2US3A6OX6WIGW3XYQ, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.957067453 (89h49m23.828150528s) [noise=true , address=81.171.14.70:4031 , piece_id=TL65276T2XDWZUTNUZ7IBI5D4BSAMB5S6W6G5CT3NZ6JF2WSESOA, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.958192827 (89h49m23.829275862s) [noise=true , address=185.167.204.205:4015 , piece_id=YU77JIVCOQ37G52QR4RKZTQ2WY7ME4JPHBM7RTUHBHAM5A4DCFDQ, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.963154113 (89h49m23.834237159s) [noise=true , address=185.167.204.206:4065 , piece_id=LQIMSCI7CLY57EOLIERM5NSR64QSNR64FD6H2L6OUQV6VANBP77Q, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.964026705 (89h49m23.83510977s) [noise=true , address=185.167.204.206:4057 , piece_id=AVPMO43TLI7KCC52Y4OH2W2G3RLMGLBU7BTGPEO5FG2VUN7272DA, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.967189518 (89h49m23.838272563s) [noise=true , address=81.171.14.70:4030 , piece_id=6VDC74RCXTDZDGRMAGDR6HKRMEZAUCVM2KXIORPRFHOUPZ4SRURQ, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.969161493 (89h49m23.840244548s) [noise=true , address=89.149.201.147:4019 , piece_id=JI735NFBOCR6YSMRDKNIH653LHV3D5LGY34PJMPISBZE7TV57MCA, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.969242788 (89h49m23.840325834s) [noise=true , address=185.167.204.206:4060 , piece_id=TR5JMA65VL3X2JYKINUGVORZFKKBKV2NY2LBLHPQJS5Z4P2NJHWA, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.970553794 (89h49m23.841636849s) [noise=true , address=185.167.204.205:4016 , piece_id=T4SMB72E6JS76626B3SOUTKCFK6OHRPEQSKOLCBBQW5XY5PO3TNA, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.972813026 (89h49m23.843896081s) [noise=true , address=185.167.204.206:4054 , piece_id=CBV3DDAZUBWN4RVNVA37BC5ESANQPT4MYGHWMNTO4AZJ32ZZ7LWQ, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.978089925 (89h49m23.84917298s) [noise=true , address=185.167.204.205:4036 , piece_id=JE2RTZWLPINK4DZEIN5VZSF7NQNDIJ4EEEHLNXIGCXZAVVK23RGQ, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:24.785720741 (89h49m24.656803786s) [upload=78358]: Next returned nil segment Dec 9 09:18:24.785736831 (89h49m24.656819876s) [upload=78358]: Waiting for error group managing segments... Dec 9 09:18:25.168467403 (89h49m25.039550448s) [noise=true , address=185.167.204.206:4049 , piece_id=MEZGGAN2ZFVW535D3QU5FIF5X2XNI67T7FMNIJQMMXLHF5NN7KCA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.185302831 (89h49m25.056385876s) [noise=true , address=185.167.204.206:4054 , piece_id=CBV3DDAZUBWN4RVNVA37BC5ESANQPT4MYGHWMNTO4AZJ32ZZ7LWQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.185793916 (89h49m25.056876961s) [noise=true , address=185.167.204.206:4057 , piece_id=AVPMO43TLI7KCC52Y4OH2W2G3RLMGLBU7BTGPEO5FG2VUN7272DA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.186032270 (89h49m25.057115315s) [noise=true , address=185.167.204.206:4050 , piece_id=LBHTVVRMFZERIJPYLJGIZB6FC4YERLQI4YL2OMUXKN37OLQHDHDA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.188698217 (89h49m25.059781262s) [noise=true , address=185.167.204.205:4016 , piece_id=T4SMB72E6JS76626B3SOUTKCFK6OHRPEQSKOLCBBQW5XY5PO3TNA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.198062969 (89h49m25.069146014s) [noise=true , address=185.167.204.205:4024 , piece_id=Q53WJVADCT7IWGNY7OUXAA4VGI6ARFLTGDLM2KM6LBE2HBBA3DAQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.201189182 (89h49m25.072272227s) [noise=true , address=185.167.204.206:4060 , piece_id=TR5JMA65VL3X2JYKINUGVORZFKKBKV2NY2LBLHPQJS5Z4P2NJHWA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.202370381 (89h49m25.073453426s) [noise=true , address=185.167.204.206:4069 , piece_id=LD44HDOWUF7N2LWP3MU4DSDXO3YAI6KVCUP2US3A6OX6WIGW3XYQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.205089780 (89h49m25.076172816s) [noise=true , address=185.167.204.205:4032 , piece_id=R75QHC7MZRLI6HXHLCBWU5ZCKUQQI35I6URWVG7WKDIAFWIPN7XQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.207830660 (89h49m25.078913705s) [noise=true , address=185.167.204.206:4037 , piece_id=NW4QWZFM7JA6UF7JNUWNPEYL76BLWLB5KF3VLHGJZSCORTKL5SOA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.208925113 (89h49m25.080008159s) [noise=true , address=185.167.204.205:4036 , piece_id=JE2RTZWLPINK4DZEIN5VZSF7NQNDIJ4EEEHLNXIGCXZAVVK23RGQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.209472386 (89h49m25.080555431s) [noise=true , address=185.167.204.206:4065 , piece_id=LQIMSCI7CLY57EOLIERM5NSR64QSNR64FD6H2L6OUQV6VANBP77Q, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.211577764 (89h49m25.082660809s) [noise=true , address=185.167.204.205:4017 , piece_id=MILTVCWHWQJBMZFZPFXXCDSTPWRYHDHBFQEOPO6YG5PQTMD3RONQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.219514148 (89h49m25.090597183s) [noise=true , address=185.167.204.205:4015 , piece_id=YU77JIVCOQ37G52QR4RKZTQ2WY7ME4JPHBM7RTUHBHAM5A4DCFDQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.769337000 (89h49m25.640420045s) [noise=true , address=81.171.14.70:4039 , piece_id=QFA6JIAMTETMRGCCK6PJDUKWH43PZJOQQ4X5Y5ILCGL5R37OYTUQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.772473423 (89h49m25.643556468s) [noise=true , address=81.171.14.70:4032 , piece_id=FJAX2O76ZV2L5FWWXABF7FGT5JUQ6A3YDMKJUQSREBEJPIGQP7RA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.800364573 (89h49m25.671447608s) [noise=true , address=81.171.14.70:4030 , piece_id=6VDC74RCXTDZDGRMAGDR6HKRMEZAUCVM2KXIORPRFHOUPZ4SRURQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.800677198 (89h49m25.671760243s) [noise=true , address=81.171.14.70:4047 , piece_id=WZJLBCHCHZHOHL5IC66ZWGIF4EOHNOEH3XBVBXIZCV65YRSTNPBA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.810545961 (89h49m25.681629006s) [noise=true , address=81.171.14.70:4046 , piece_id=CZRVAU2PNZBUSVCYX5MOV72SZXID4GPJ7BDPPD4NIJ235EY62YMA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.838216351 (89h49m25.709299396s) [noise=true , address=81.171.14.70:4031 , piece_id=TL65276T2XDWZUTNUZ7IBI5D4BSAMB5S6W6G5CT3NZ6JF2WSESOA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:26.250350354 (89h49m26.121433399s) [noise=true , address=81.171.14.70:4036 , piece_id=3PRKMFHXBCM52IQ3IUGSQR5G727WYKUTVJBVC5QERPZ2WFCGSA3Q, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:26.277784646 (89h49m26.148867691s) [noise=true , address=81.171.14.70:4048 , piece_id=PTOL6CIPLQS6LTCCLI6PFEXOAOXHTMUQ4SYTRRJRWSXTED4VHMFQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:26.519380506 (89h49m26.390463552s) [noise=true , address=89.149.201.147:4005 , piece_id=HVHGMPYGLWAMDJVZTNUVXZ3GCPWDKDF3SBCJO2ILKKQGSPAYEBFA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:26.910352793 (89h49m26.781435828s) [noise=true , address=89.149.201.147:4001 , piece_id=PJ3CG6DO2HMSDREM23CEEBEDEEMREOBD2C2WMZSM2B5MSRASY6GQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:26.962433886 (89h49m26.833516921s) [noise=true , address=89.149.201.147:4004 , piece_id=JNEKFVLJN5FTXT46CVFQVGOA5BTDNL6TDB655V6WWMZWT5LC7FBQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:26.974479733 (89h49m26.845562768s) [noise=true , address=89.149.201.147:4023 , piece_id=LQPBNSGESUY2CONNWGJWPE67S76QATD3T2DICJPYJ3H4J2NRCWKQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:26.974507296 (89h49m26.845590341s) [seg_pos={0 0}, upload=78358]: Segment reached optimal threshold of 26 pieces. Dec 9 09:18:26.974751972 (89h49m26.845835017s) [noise=true , address=89.149.201.147:4007 , piece_id=PY2KA7ZSI422BIOOTFWMLFY3LVQIFS4DC2VWTBR5HUN5C36DR53Q, seg_pos={0 0}, upload=78358]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew): protocol: expected piece hash; context canceled; EOF Dec 9 09:18:26.974762383 (89h49m26.845845438s) [noise=true , address=89.149.201.147:4007 , piece_id=PY2KA7ZSI422BIOOTFWMLFY3LVQIFS4DC2VWTBR5HUN5C36DR53Q, seg_pos={0 0}, upload=78358]: Long tail cancellation due to optimal threshold Dec 9 09:18:26.974851171 (89h49m26.845934226s) [noise=true , address=89.149.201.147:4019 , piece_id=JI735NFBOCR6YSMRDKNIH653LHV3D5LGY34PJMPISBZE7TV57MCA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wgZQR7Ezekmvf9cNwCZ8CmZBjZgoqWC3N2UE3n4YQSmBsqQQ2): protocol: expected piece hash; context canceled; EOF Dec 9 09:18:26.974873604 (89h49m26.845956669s) [noise=true , address=89.149.201.147:4019 , piece_id=JI735NFBOCR6YSMRDKNIH653LHV3D5LGY34PJMPISBZE7TV57MCA, seg_pos={0 0}, upload=78358]: Long tail cancellation due to optimal threshold Dec 9 09:18:26.974923419 (89h49m26.846006464s) [seg_pos={0 0}, upload=78358]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 09:18:26.975251043 (89h49m26.846334138s) [upload=78358]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 09:18:26.975262835 (89h49m26.84634587s) [upload=78358]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 09:19:23.276531408 (89h50m23.147614453s) [upload=78358]: Done uploading segments... Dec 9 09:19:23.276661235 (89h50m23.14774428s) [upload=78358]: Upload finished. err: metaclient: context deadline exceeded Duration: 17.589189834s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......X.....................................................] duration: 3.417987207s trace: https://jaeger.dev.storj.io/trace/62479de42d709d4f FAIL (2025-12-09 09:51:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.418040749s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [..................................X...X..............X......] duration: 5.887744738s trace: https://jaeger.dev.storj.io/trace/9ef0062b2b9b30b FAIL (2025-12-09 09:51:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.887772209s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [X......X....................................................] duration: 3.524574612s trace: https://jaeger.dev.storj.io/trace/62479de42d709d4f FAIL (2025-12-09 09:51:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.524602204s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 684.286764ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/754b20513c76ffd8 FAIL (2025-12-09 09:50:55 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 684.317382ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 1.91850413s trace: https://jaeger.dev.storj.io/trace/92f373d508d696c FAIL (2025-12-09 09:50:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 09:26:29.076085885 (89h57m28.94716892s) [upload=78482]: Starting upload Dec 9 09:26:29.076139197 (89h57m28.947222232s) [upload=78482]: Done starting upload Dec 9 09:26:29.076210933 (89h57m28.947293978s) [upload=78482]: Uploading segments... Dec 9 09:26:29.076295604 (89h57m28.947378639s) [upload=78482]: Got next segment. Inline: true Dec 9 09:26:29.076303269 (89h57m28.947386314s) [upload=78482]: Waiting for error group managing segments... Dec 9 09:26:29.076399502 (89h57m28.947482537s) [upload=78482]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 09:26:29.076427595 (89h57m28.94751064s) [upload=78482]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 9 09:26:29.076432515 (89h57m28.94751555s) [upload=78482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 09:27:28.400202146 (89h58m28.271285192s) [upload=78482]: Done uploading segments... Dec 9 09:27:28.400236482 (89h58m28.271319517s) [upload=78482]: Upload finished. err: metaclient: context deadline exceeded Duration:* 1.918534729s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 8.966926479s trace: https://jaeger.dev.storj.io/trace/2278184cf7cb09d3 FAIL (2025-12-09 09:50:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== HDOWUF7N2LWP3MU4DSDXO3YAI6KVCUP2US3A6OX6WIGW3XYQ, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.957067453 (89h49m23.828150528s) [noise=true , address=81.171.14.70:4031 , piece_id=TL65276T2XDWZUTNUZ7IBI5D4BSAMB5S6W6G5CT3NZ6JF2WSESOA, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.958192827 (89h49m23.829275862s) [noise=true , address=185.167.204.205:4015 , piece_id=YU77JIVCOQ37G52QR4RKZTQ2WY7ME4JPHBM7RTUHBHAM5A4DCFDQ, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.963154113 (89h49m23.834237159s) [noise=true , address=185.167.204.206:4065 , piece_id=LQIMSCI7CLY57EOLIERM5NSR64QSNR64FD6H2L6OUQV6VANBP77Q, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.964026705 (89h49m23.83510977s) [noise=true , address=185.167.204.206:4057 , piece_id=AVPMO43TLI7KCC52Y4OH2W2G3RLMGLBU7BTGPEO5FG2VUN7272DA, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.967189518 (89h49m23.838272563s) [noise=true , address=81.171.14.70:4030 , piece_id=6VDC74RCXTDZDGRMAGDR6HKRMEZAUCVM2KXIORPRFHOUPZ4SRURQ, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.969161493 (89h49m23.840244548s) [noise=true , address=89.149.201.147:4019 , piece_id=JI735NFBOCR6YSMRDKNIH653LHV3D5LGY34PJMPISBZE7TV57MCA, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.969242788 (89h49m23.840325834s) [noise=true , address=185.167.204.206:4060 , piece_id=TR5JMA65VL3X2JYKINUGVORZFKKBKV2NY2LBLHPQJS5Z4P2NJHWA, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.970553794 (89h49m23.841636849s) [noise=true , address=185.167.204.205:4016 , piece_id=T4SMB72E6JS76626B3SOUTKCFK6OHRPEQSKOLCBBQW5XY5PO3TNA, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.972813026 (89h49m23.843896081s) [noise=true , address=185.167.204.206:4054 , piece_id=CBV3DDAZUBWN4RVNVA37BC5ESANQPT4MYGHWMNTO4AZJ32ZZ7LWQ, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.978089925 (89h49m23.84917298s) [noise=true , address=185.167.204.205:4036 , piece_id=JE2RTZWLPINK4DZEIN5VZSF7NQNDIJ4EEEHLNXIGCXZAVVK23RGQ, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:24.785720741 (89h49m24.656803786s) [upload=78358]: Next returned nil segment Dec 9 09:18:24.785736831 (89h49m24.656819876s) [upload=78358]: Waiting for error group managing segments... Dec 9 09:18:25.168467403 (89h49m25.039550448s) [noise=true , address=185.167.204.206:4049 , piece_id=MEZGGAN2ZFVW535D3QU5FIF5X2XNI67T7FMNIJQMMXLHF5NN7KCA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.185302831 (89h49m25.056385876s) [noise=true , address=185.167.204.206:4054 , piece_id=CBV3DDAZUBWN4RVNVA37BC5ESANQPT4MYGHWMNTO4AZJ32ZZ7LWQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.185793916 (89h49m25.056876961s) [noise=true , address=185.167.204.206:4057 , piece_id=AVPMO43TLI7KCC52Y4OH2W2G3RLMGLBU7BTGPEO5FG2VUN7272DA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.186032270 (89h49m25.057115315s) [noise=true , address=185.167.204.206:4050 , piece_id=LBHTVVRMFZERIJPYLJGIZB6FC4YERLQI4YL2OMUXKN37OLQHDHDA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.188698217 (89h49m25.059781262s) [noise=true , address=185.167.204.205:4016 , piece_id=T4SMB72E6JS76626B3SOUTKCFK6OHRPEQSKOLCBBQW5XY5PO3TNA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.198062969 (89h49m25.069146014s) [noise=true , address=185.167.204.205:4024 , piece_id=Q53WJVADCT7IWGNY7OUXAA4VGI6ARFLTGDLM2KM6LBE2HBBA3DAQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.201189182 (89h49m25.072272227s) [noise=true , address=185.167.204.206:4060 , piece_id=TR5JMA65VL3X2JYKINUGVORZFKKBKV2NY2LBLHPQJS5Z4P2NJHWA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.202370381 (89h49m25.073453426s) [noise=true , address=185.167.204.206:4069 , piece_id=LD44HDOWUF7N2LWP3MU4DSDXO3YAI6KVCUP2US3A6OX6WIGW3XYQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.205089780 (89h49m25.076172816s) [noise=true , address=185.167.204.205:4032 , piece_id=R75QHC7MZRLI6HXHLCBWU5ZCKUQQI35I6URWVG7WKDIAFWIPN7XQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.207830660 (89h49m25.078913705s) [noise=true , address=185.167.204.206:4037 , piece_id=NW4QWZFM7JA6UF7JNUWNPEYL76BLWLB5KF3VLHGJZSCORTKL5SOA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.208925113 (89h49m25.080008159s) [noise=true , address=185.167.204.205:4036 , piece_id=JE2RTZWLPINK4DZEIN5VZSF7NQNDIJ4EEEHLNXIGCXZAVVK23RGQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.209472386 (89h49m25.080555431s) [noise=true , address=185.167.204.206:4065 , piece_id=LQIMSCI7CLY57EOLIERM5NSR64QSNR64FD6H2L6OUQV6VANBP77Q, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.211577764 (89h49m25.082660809s) [noise=true , address=185.167.204.205:4017 , piece_id=MILTVCWHWQJBMZFZPFXXCDSTPWRYHDHBFQEOPO6YG5PQTMD3RONQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.219514148 (89h49m25.090597183s) [noise=true , address=185.167.204.205:4015 , piece_id=YU77JIVCOQ37G52QR4RKZTQ2WY7ME4JPHBM7RTUHBHAM5A4DCFDQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.769337000 (89h49m25.640420045s) [noise=true , address=81.171.14.70:4039 , piece_id=QFA6JIAMTETMRGCCK6PJDUKWH43PZJOQQ4X5Y5ILCGL5R37OYTUQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.772473423 (89h49m25.643556468s) [noise=true , address=81.171.14.70:4032 , piece_id=FJAX2O76ZV2L5FWWXABF7FGT5JUQ6A3YDMKJUQSREBEJPIGQP7RA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.800364573 (89h49m25.671447608s) [noise=true , address=81.171.14.70:4030 , piece_id=6VDC74RCXTDZDGRMAGDR6HKRMEZAUCVM2KXIORPRFHOUPZ4SRURQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.800677198 (89h49m25.671760243s) [noise=true , address=81.171.14.70:4047 , piece_id=WZJLBCHCHZHOHL5IC66ZWGIF4EOHNOEH3XBVBXIZCV65YRSTNPBA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.810545961 (89h49m25.681629006s) [noise=true , address=81.171.14.70:4046 , piece_id=CZRVAU2PNZBUSVCYX5MOV72SZXID4GPJ7BDPPD4NIJ235EY62YMA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.838216351 (89h49m25.709299396s) [noise=true , address=81.171.14.70:4031 , piece_id=TL65276T2XDWZUTNUZ7IBI5D4BSAMB5S6W6G5CT3NZ6JF2WSESOA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:26.250350354 (89h49m26.121433399s) [noise=true , address=81.171.14.70:4036 , piece_id=3PRKMFHXBCM52IQ3IUGSQR5G727WYKUTVJBVC5QERPZ2WFCGSA3Q, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:26.277784646 (89h49m26.148867691s) [noise=true , address=81.171.14.70:4048 , piece_id=PTOL6CIPLQS6LTCCLI6PFEXOAOXHTMUQ4SYTRRJRWSXTED4VHMFQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:26.519380506 (89h49m26.390463552s) [noise=true , address=89.149.201.147:4005 , piece_id=HVHGMPYGLWAMDJVZTNUVXZ3GCPWDKDF3SBCJO2ILKKQGSPAYEBFA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:26.910352793 (89h49m26.781435828s) [noise=true , address=89.149.201.147:4001 , piece_id=PJ3CG6DO2HMSDREM23CEEBEDEEMREOBD2C2WMZSM2B5MSRASY6GQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:26.962433886 (89h49m26.833516921s) [noise=true , address=89.149.201.147:4004 , piece_id=JNEKFVLJN5FTXT46CVFQVGOA5BTDNL6TDB655V6WWMZWT5LC7FBQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:26.974479733 (89h49m26.845562768s) [noise=true , address=89.149.201.147:4023 , piece_id=LQPBNSGESUY2CONNWGJWPE67S76QATD3T2DICJPYJ3H4J2NRCWKQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:26.974507296 (89h49m26.845590341s) [seg_pos={0 0}, upload=78358]: Segment reached optimal threshold of 26 pieces. Dec 9 09:18:26.974751972 (89h49m26.845835017s) [noise=true , address=89.149.201.147:4007 , piece_id=PY2KA7ZSI422BIOOTFWMLFY3LVQIFS4DC2VWTBR5HUN5C36DR53Q, seg_pos={0 0}, upload=78358]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew): protocol: expected piece hash; context canceled; EOF Dec 9 09:18:26.974762383 (89h49m26.845845438s) [noise=true , address=89.149.201.147:4007 , piece_id=PY2KA7ZSI422BIOOTFWMLFY3LVQIFS4DC2VWTBR5HUN5C36DR53Q, seg_pos={0 0}, upload=78358]: Long tail cancellation due to optimal threshold Dec 9 09:18:26.974851171 (89h49m26.845934226s) [noise=true , address=89.149.201.147:4019 , piece_id=JI735NFBOCR6YSMRDKNIH653LHV3D5LGY34PJMPISBZE7TV57MCA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wgZQR7Ezekmvf9cNwCZ8CmZBjZgoqWC3N2UE3n4YQSmBsqQQ2): protocol: expected piece hash; context canceled; EOF Dec 9 09:18:26.974873604 (89h49m26.845956669s) [noise=true , address=89.149.201.147:4019 , piece_id=JI735NFBOCR6YSMRDKNIH653LHV3D5LGY34PJMPISBZE7TV57MCA, seg_pos={0 0}, upload=78358]: Long tail cancellation due to optimal threshold Dec 9 09:18:26.974923419 (89h49m26.846006464s) [seg_pos={0 0}, upload=78358]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 09:18:26.975251043 (89h49m26.846334138s) [upload=78358]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 09:18:26.975262835 (89h49m26.84634587s) [upload=78358]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 09:19:23.276531408 (89h50m23.147614453s) [upload=78358]: Done uploading segments... Dec 9 09:19:23.276661235 (89h50m23.14774428s) [upload=78358]: Upload finished. err: metaclient: context deadline exceeded Duration: 8.966957708s core uplink eu-select-1 eu1-placement-15
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 681.188803ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/754b20513c76ffd8 FAIL (2025-12-09 09:50:28 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 681.221316ms core satellite us1
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 690.970321ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/754b20513c76ffd8 FAIL (2025-12-09 09:50:11 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 690.997953ms core satellite us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [...................................X...X..............X.....] duration: 5.729903915s trace: https://jaeger.dev.storj.io/trace/9ef0062b2b9b30b FAIL (2025-12-09 09:50:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.736946326s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.X......X...................................................] duration: 3.598850049s trace: https://jaeger.dev.storj.io/trace/62479de42d709d4f FAIL (2025-12-09 09:49:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.598996197s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 11.84262442s trace: https://jaeger.dev.storj.io/trace/2278184cf7cb09d3 FAIL (2025-12-09 09:49:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== HDOWUF7N2LWP3MU4DSDXO3YAI6KVCUP2US3A6OX6WIGW3XYQ, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.957067453 (89h49m23.828150528s) [noise=true , address=81.171.14.70:4031 , piece_id=TL65276T2XDWZUTNUZ7IBI5D4BSAMB5S6W6G5CT3NZ6JF2WSESOA, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.958192827 (89h49m23.829275862s) [noise=true , address=185.167.204.205:4015 , piece_id=YU77JIVCOQ37G52QR4RKZTQ2WY7ME4JPHBM7RTUHBHAM5A4DCFDQ, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.963154113 (89h49m23.834237159s) [noise=true , address=185.167.204.206:4065 , piece_id=LQIMSCI7CLY57EOLIERM5NSR64QSNR64FD6H2L6OUQV6VANBP77Q, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.964026705 (89h49m23.83510977s) [noise=true , address=185.167.204.206:4057 , piece_id=AVPMO43TLI7KCC52Y4OH2W2G3RLMGLBU7BTGPEO5FG2VUN7272DA, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.967189518 (89h49m23.838272563s) [noise=true , address=81.171.14.70:4030 , piece_id=6VDC74RCXTDZDGRMAGDR6HKRMEZAUCVM2KXIORPRFHOUPZ4SRURQ, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.969161493 (89h49m23.840244548s) [noise=true , address=89.149.201.147:4019 , piece_id=JI735NFBOCR6YSMRDKNIH653LHV3D5LGY34PJMPISBZE7TV57MCA, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.969242788 (89h49m23.840325834s) [noise=true , address=185.167.204.206:4060 , piece_id=TR5JMA65VL3X2JYKINUGVORZFKKBKV2NY2LBLHPQJS5Z4P2NJHWA, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.970553794 (89h49m23.841636849s) [noise=true , address=185.167.204.205:4016 , piece_id=T4SMB72E6JS76626B3SOUTKCFK6OHRPEQSKOLCBBQW5XY5PO3TNA, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.972813026 (89h49m23.843896081s) [noise=true , address=185.167.204.206:4054 , piece_id=CBV3DDAZUBWN4RVNVA37BC5ESANQPT4MYGHWMNTO4AZJ32ZZ7LWQ, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.978089925 (89h49m23.84917298s) [noise=true , address=185.167.204.205:4036 , piece_id=JE2RTZWLPINK4DZEIN5VZSF7NQNDIJ4EEEHLNXIGCXZAVVK23RGQ, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:24.785720741 (89h49m24.656803786s) [upload=78358]: Next returned nil segment Dec 9 09:18:24.785736831 (89h49m24.656819876s) [upload=78358]: Waiting for error group managing segments... Dec 9 09:18:25.168467403 (89h49m25.039550448s) [noise=true , address=185.167.204.206:4049 , piece_id=MEZGGAN2ZFVW535D3QU5FIF5X2XNI67T7FMNIJQMMXLHF5NN7KCA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.185302831 (89h49m25.056385876s) [noise=true , address=185.167.204.206:4054 , piece_id=CBV3DDAZUBWN4RVNVA37BC5ESANQPT4MYGHWMNTO4AZJ32ZZ7LWQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.185793916 (89h49m25.056876961s) [noise=true , address=185.167.204.206:4057 , piece_id=AVPMO43TLI7KCC52Y4OH2W2G3RLMGLBU7BTGPEO5FG2VUN7272DA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.186032270 (89h49m25.057115315s) [noise=true , address=185.167.204.206:4050 , piece_id=LBHTVVRMFZERIJPYLJGIZB6FC4YERLQI4YL2OMUXKN37OLQHDHDA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.188698217 (89h49m25.059781262s) [noise=true , address=185.167.204.205:4016 , piece_id=T4SMB72E6JS76626B3SOUTKCFK6OHRPEQSKOLCBBQW5XY5PO3TNA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.198062969 (89h49m25.069146014s) [noise=true , address=185.167.204.205:4024 , piece_id=Q53WJVADCT7IWGNY7OUXAA4VGI6ARFLTGDLM2KM6LBE2HBBA3DAQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.201189182 (89h49m25.072272227s) [noise=true , address=185.167.204.206:4060 , piece_id=TR5JMA65VL3X2JYKINUGVORZFKKBKV2NY2LBLHPQJS5Z4P2NJHWA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.202370381 (89h49m25.073453426s) [noise=true , address=185.167.204.206:4069 , piece_id=LD44HDOWUF7N2LWP3MU4DSDXO3YAI6KVCUP2US3A6OX6WIGW3XYQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.205089780 (89h49m25.076172816s) [noise=true , address=185.167.204.205:4032 , piece_id=R75QHC7MZRLI6HXHLCBWU5ZCKUQQI35I6URWVG7WKDIAFWIPN7XQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.207830660 (89h49m25.078913705s) [noise=true , address=185.167.204.206:4037 , piece_id=NW4QWZFM7JA6UF7JNUWNPEYL76BLWLB5KF3VLHGJZSCORTKL5SOA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.208925113 (89h49m25.080008159s) [noise=true , address=185.167.204.205:4036 , piece_id=JE2RTZWLPINK4DZEIN5VZSF7NQNDIJ4EEEHLNXIGCXZAVVK23RGQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.209472386 (89h49m25.080555431s) [noise=true , address=185.167.204.206:4065 , piece_id=LQIMSCI7CLY57EOLIERM5NSR64QSNR64FD6H2L6OUQV6VANBP77Q, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.211577764 (89h49m25.082660809s) [noise=true , address=185.167.204.205:4017 , piece_id=MILTVCWHWQJBMZFZPFXXCDSTPWRYHDHBFQEOPO6YG5PQTMD3RONQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.219514148 (89h49m25.090597183s) [noise=true , address=185.167.204.205:4015 , piece_id=YU77JIVCOQ37G52QR4RKZTQ2WY7ME4JPHBM7RTUHBHAM5A4DCFDQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.769337000 (89h49m25.640420045s) [noise=true , address=81.171.14.70:4039 , piece_id=QFA6JIAMTETMRGCCK6PJDUKWH43PZJOQQ4X5Y5ILCGL5R37OYTUQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.772473423 (89h49m25.643556468s) [noise=true , address=81.171.14.70:4032 , piece_id=FJAX2O76ZV2L5FWWXABF7FGT5JUQ6A3YDMKJUQSREBEJPIGQP7RA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.800364573 (89h49m25.671447608s) [noise=true , address=81.171.14.70:4030 , piece_id=6VDC74RCXTDZDGRMAGDR6HKRMEZAUCVM2KXIORPRFHOUPZ4SRURQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.800677198 (89h49m25.671760243s) [noise=true , address=81.171.14.70:4047 , piece_id=WZJLBCHCHZHOHL5IC66ZWGIF4EOHNOEH3XBVBXIZCV65YRSTNPBA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.810545961 (89h49m25.681629006s) [noise=true , address=81.171.14.70:4046 , piece_id=CZRVAU2PNZBUSVCYX5MOV72SZXID4GPJ7BDPPD4NIJ235EY62YMA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.838216351 (89h49m25.709299396s) [noise=true , address=81.171.14.70:4031 , piece_id=TL65276T2XDWZUTNUZ7IBI5D4BSAMB5S6W6G5CT3NZ6JF2WSESOA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:26.250350354 (89h49m26.121433399s) [noise=true , address=81.171.14.70:4036 , piece_id=3PRKMFHXBCM52IQ3IUGSQR5G727WYKUTVJBVC5QERPZ2WFCGSA3Q, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:26.277784646 (89h49m26.148867691s) [noise=true , address=81.171.14.70:4048 , piece_id=PTOL6CIPLQS6LTCCLI6PFEXOAOXHTMUQ4SYTRRJRWSXTED4VHMFQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:26.519380506 (89h49m26.390463552s) [noise=true , address=89.149.201.147:4005 , piece_id=HVHGMPYGLWAMDJVZTNUVXZ3GCPWDKDF3SBCJO2ILKKQGSPAYEBFA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:26.910352793 (89h49m26.781435828s) [noise=true , address=89.149.201.147:4001 , piece_id=PJ3CG6DO2HMSDREM23CEEBEDEEMREOBD2C2WMZSM2B5MSRASY6GQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:26.962433886 (89h49m26.833516921s) [noise=true , address=89.149.201.147:4004 , piece_id=JNEKFVLJN5FTXT46CVFQVGOA5BTDNL6TDB655V6WWMZWT5LC7FBQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:26.974479733 (89h49m26.845562768s) [noise=true , address=89.149.201.147:4023 , piece_id=LQPBNSGESUY2CONNWGJWPE67S76QATD3T2DICJPYJ3H4J2NRCWKQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:26.974507296 (89h49m26.845590341s) [seg_pos={0 0}, upload=78358]: Segment reached optimal threshold of 26 pieces. Dec 9 09:18:26.974751972 (89h49m26.845835017s) [noise=true , address=89.149.201.147:4007 , piece_id=PY2KA7ZSI422BIOOTFWMLFY3LVQIFS4DC2VWTBR5HUN5C36DR53Q, seg_pos={0 0}, upload=78358]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew): protocol: expected piece hash; context canceled; EOF Dec 9 09:18:26.974762383 (89h49m26.845845438s) [noise=true , address=89.149.201.147:4007 , piece_id=PY2KA7ZSI422BIOOTFWMLFY3LVQIFS4DC2VWTBR5HUN5C36DR53Q, seg_pos={0 0}, upload=78358]: Long tail cancellation due to optimal threshold Dec 9 09:18:26.974851171 (89h49m26.845934226s) [noise=true , address=89.149.201.147:4019 , piece_id=JI735NFBOCR6YSMRDKNIH653LHV3D5LGY34PJMPISBZE7TV57MCA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wgZQR7Ezekmvf9cNwCZ8CmZBjZgoqWC3N2UE3n4YQSmBsqQQ2): protocol: expected piece hash; context canceled; EOF Dec 9 09:18:26.974873604 (89h49m26.845956669s) [noise=true , address=89.149.201.147:4019 , piece_id=JI735NFBOCR6YSMRDKNIH653LHV3D5LGY34PJMPISBZE7TV57MCA, seg_pos={0 0}, upload=78358]: Long tail cancellation due to optimal threshold Dec 9 09:18:26.974923419 (89h49m26.846006464s) [seg_pos={0 0}, upload=78358]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 09:18:26.975251043 (89h49m26.846334138s) [upload=78358]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 09:18:26.975262835 (89h49m26.84634587s) [upload=78358]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 09:19:23.276531408 (89h50m23.147614453s) [upload=78358]: Done uploading segments... Dec 9 09:19:23.276661235 (89h50m23.14774428s) [upload=78358]: Upload finished. err: metaclient: context deadline exceeded Duration: 11.842641133s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 1.945016056s trace: https://jaeger.dev.storj.io/trace/92f373d508d696c FAIL (2025-12-09 09:49:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 09:26:29.076085885 (89h57m28.94716892s) [upload=78482]: Starting upload Dec 9 09:26:29.076139197 (89h57m28.947222232s) [upload=78482]: Done starting upload Dec 9 09:26:29.076210933 (89h57m28.947293978s) [upload=78482]: Uploading segments... Dec 9 09:26:29.076295604 (89h57m28.947378639s) [upload=78482]: Got next segment. Inline: true Dec 9 09:26:29.076303269 (89h57m28.947386314s) [upload=78482]: Waiting for error group managing segments... Dec 9 09:26:29.076399502 (89h57m28.947482537s) [upload=78482]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 09:26:29.076427595 (89h57m28.94751064s) [upload=78482]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 9 09:26:29.076432515 (89h57m28.94751555s) [upload=78482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 09:27:28.400202146 (89h58m28.271285192s) [upload=78482]: Done uploading segments... Dec 9 09:27:28.400236482 (89h58m28.271319517s) [upload=78482]: Upload finished. err: metaclient: context deadline exceeded Duration:* 1.945047295s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 696.868963ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/754b20513c76ffd8 FAIL (2025-12-09 09:49:36 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 696.892628ms core satellite us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..X......X..................................................] duration: 3.464489368s trace: https://jaeger.dev.storj.io/trace/62479de42d709d4f FAIL (2025-12-09 09:49:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.464519145s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [....................................X...X..............X....] duration: 5.545896288s trace: https://jaeger.dev.storj.io/trace/9ef0062b2b9b30b FAIL (2025-12-09 09:49:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.546002139s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 1.968213438s trace: https://jaeger.dev.storj.io/trace/92f373d508d696c FAIL (2025-12-09 09:49:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 09:26:29.076085885 (89h57m28.94716892s) [upload=78482]: Starting upload Dec 9 09:26:29.076139197 (89h57m28.947222232s) [upload=78482]: Done starting upload Dec 9 09:26:29.076210933 (89h57m28.947293978s) [upload=78482]: Uploading segments... Dec 9 09:26:29.076295604 (89h57m28.947378639s) [upload=78482]: Got next segment. Inline: true Dec 9 09:26:29.076303269 (89h57m28.947386314s) [upload=78482]: Waiting for error group managing segments... Dec 9 09:26:29.076399502 (89h57m28.947482537s) [upload=78482]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 09:26:29.076427595 (89h57m28.94751064s) [upload=78482]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 9 09:26:29.076432515 (89h57m28.94751555s) [upload=78482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 09:27:28.400202146 (89h58m28.271285192s) [upload=78482]: Done uploading segments... Dec 9 09:27:28.400236482 (89h58m28.271319517s) [upload=78482]: Upload finished. err: metaclient: context deadline exceeded Duration:* 1.968243928s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 13.339340073s trace: https://jaeger.dev.storj.io/trace/2278184cf7cb09d3 FAIL (2025-12-09 09:49:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== HDOWUF7N2LWP3MU4DSDXO3YAI6KVCUP2US3A6OX6WIGW3XYQ, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.957067453 (89h49m23.828150528s) [noise=true , address=81.171.14.70:4031 , piece_id=TL65276T2XDWZUTNUZ7IBI5D4BSAMB5S6W6G5CT3NZ6JF2WSESOA, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.958192827 (89h49m23.829275862s) [noise=true , address=185.167.204.205:4015 , piece_id=YU77JIVCOQ37G52QR4RKZTQ2WY7ME4JPHBM7RTUHBHAM5A4DCFDQ, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.963154113 (89h49m23.834237159s) [noise=true , address=185.167.204.206:4065 , piece_id=LQIMSCI7CLY57EOLIERM5NSR64QSNR64FD6H2L6OUQV6VANBP77Q, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.964026705 (89h49m23.83510977s) [noise=true , address=185.167.204.206:4057 , piece_id=AVPMO43TLI7KCC52Y4OH2W2G3RLMGLBU7BTGPEO5FG2VUN7272DA, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.967189518 (89h49m23.838272563s) [noise=true , address=81.171.14.70:4030 , piece_id=6VDC74RCXTDZDGRMAGDR6HKRMEZAUCVM2KXIORPRFHOUPZ4SRURQ, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.969161493 (89h49m23.840244548s) [noise=true , address=89.149.201.147:4019 , piece_id=JI735NFBOCR6YSMRDKNIH653LHV3D5LGY34PJMPISBZE7TV57MCA, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.969242788 (89h49m23.840325834s) [noise=true , address=185.167.204.206:4060 , piece_id=TR5JMA65VL3X2JYKINUGVORZFKKBKV2NY2LBLHPQJS5Z4P2NJHWA, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.970553794 (89h49m23.841636849s) [noise=true , address=185.167.204.205:4016 , piece_id=T4SMB72E6JS76626B3SOUTKCFK6OHRPEQSKOLCBBQW5XY5PO3TNA, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.972813026 (89h49m23.843896081s) [noise=true , address=185.167.204.206:4054 , piece_id=CBV3DDAZUBWN4RVNVA37BC5ESANQPT4MYGHWMNTO4AZJ32ZZ7LWQ, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.978089925 (89h49m23.84917298s) [noise=true , address=185.167.204.205:4036 , piece_id=JE2RTZWLPINK4DZEIN5VZSF7NQNDIJ4EEEHLNXIGCXZAVVK23RGQ, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:24.785720741 (89h49m24.656803786s) [upload=78358]: Next returned nil segment Dec 9 09:18:24.785736831 (89h49m24.656819876s) [upload=78358]: Waiting for error group managing segments... Dec 9 09:18:25.168467403 (89h49m25.039550448s) [noise=true , address=185.167.204.206:4049 , piece_id=MEZGGAN2ZFVW535D3QU5FIF5X2XNI67T7FMNIJQMMXLHF5NN7KCA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.185302831 (89h49m25.056385876s) [noise=true , address=185.167.204.206:4054 , piece_id=CBV3DDAZUBWN4RVNVA37BC5ESANQPT4MYGHWMNTO4AZJ32ZZ7LWQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.185793916 (89h49m25.056876961s) [noise=true , address=185.167.204.206:4057 , piece_id=AVPMO43TLI7KCC52Y4OH2W2G3RLMGLBU7BTGPEO5FG2VUN7272DA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.186032270 (89h49m25.057115315s) [noise=true , address=185.167.204.206:4050 , piece_id=LBHTVVRMFZERIJPYLJGIZB6FC4YERLQI4YL2OMUXKN37OLQHDHDA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.188698217 (89h49m25.059781262s) [noise=true , address=185.167.204.205:4016 , piece_id=T4SMB72E6JS76626B3SOUTKCFK6OHRPEQSKOLCBBQW5XY5PO3TNA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.198062969 (89h49m25.069146014s) [noise=true , address=185.167.204.205:4024 , piece_id=Q53WJVADCT7IWGNY7OUXAA4VGI6ARFLTGDLM2KM6LBE2HBBA3DAQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.201189182 (89h49m25.072272227s) [noise=true , address=185.167.204.206:4060 , piece_id=TR5JMA65VL3X2JYKINUGVORZFKKBKV2NY2LBLHPQJS5Z4P2NJHWA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.202370381 (89h49m25.073453426s) [noise=true , address=185.167.204.206:4069 , piece_id=LD44HDOWUF7N2LWP3MU4DSDXO3YAI6KVCUP2US3A6OX6WIGW3XYQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.205089780 (89h49m25.076172816s) [noise=true , address=185.167.204.205:4032 , piece_id=R75QHC7MZRLI6HXHLCBWU5ZCKUQQI35I6URWVG7WKDIAFWIPN7XQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.207830660 (89h49m25.078913705s) [noise=true , address=185.167.204.206:4037 , piece_id=NW4QWZFM7JA6UF7JNUWNPEYL76BLWLB5KF3VLHGJZSCORTKL5SOA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.208925113 (89h49m25.080008159s) [noise=true , address=185.167.204.205:4036 , piece_id=JE2RTZWLPINK4DZEIN5VZSF7NQNDIJ4EEEHLNXIGCXZAVVK23RGQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.209472386 (89h49m25.080555431s) [noise=true , address=185.167.204.206:4065 , piece_id=LQIMSCI7CLY57EOLIERM5NSR64QSNR64FD6H2L6OUQV6VANBP77Q, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.211577764 (89h49m25.082660809s) [noise=true , address=185.167.204.205:4017 , piece_id=MILTVCWHWQJBMZFZPFXXCDSTPWRYHDHBFQEOPO6YG5PQTMD3RONQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.219514148 (89h49m25.090597183s) [noise=true , address=185.167.204.205:4015 , piece_id=YU77JIVCOQ37G52QR4RKZTQ2WY7ME4JPHBM7RTUHBHAM5A4DCFDQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.769337000 (89h49m25.640420045s) [noise=true , address=81.171.14.70:4039 , piece_id=QFA6JIAMTETMRGCCK6PJDUKWH43PZJOQQ4X5Y5ILCGL5R37OYTUQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.772473423 (89h49m25.643556468s) [noise=true , address=81.171.14.70:4032 , piece_id=FJAX2O76ZV2L5FWWXABF7FGT5JUQ6A3YDMKJUQSREBEJPIGQP7RA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.800364573 (89h49m25.671447608s) [noise=true , address=81.171.14.70:4030 , piece_id=6VDC74RCXTDZDGRMAGDR6HKRMEZAUCVM2KXIORPRFHOUPZ4SRURQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.800677198 (89h49m25.671760243s) [noise=true , address=81.171.14.70:4047 , piece_id=WZJLBCHCHZHOHL5IC66ZWGIF4EOHNOEH3XBVBXIZCV65YRSTNPBA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.810545961 (89h49m25.681629006s) [noise=true , address=81.171.14.70:4046 , piece_id=CZRVAU2PNZBUSVCYX5MOV72SZXID4GPJ7BDPPD4NIJ235EY62YMA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.838216351 (89h49m25.709299396s) [noise=true , address=81.171.14.70:4031 , piece_id=TL65276T2XDWZUTNUZ7IBI5D4BSAMB5S6W6G5CT3NZ6JF2WSESOA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:26.250350354 (89h49m26.121433399s) [noise=true , address=81.171.14.70:4036 , piece_id=3PRKMFHXBCM52IQ3IUGSQR5G727WYKUTVJBVC5QERPZ2WFCGSA3Q, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:26.277784646 (89h49m26.148867691s) [noise=true , address=81.171.14.70:4048 , piece_id=PTOL6CIPLQS6LTCCLI6PFEXOAOXHTMUQ4SYTRRJRWSXTED4VHMFQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:26.519380506 (89h49m26.390463552s) [noise=true , address=89.149.201.147:4005 , piece_id=HVHGMPYGLWAMDJVZTNUVXZ3GCPWDKDF3SBCJO2ILKKQGSPAYEBFA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:26.910352793 (89h49m26.781435828s) [noise=true , address=89.149.201.147:4001 , piece_id=PJ3CG6DO2HMSDREM23CEEBEDEEMREOBD2C2WMZSM2B5MSRASY6GQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:26.962433886 (89h49m26.833516921s) [noise=true , address=89.149.201.147:4004 , piece_id=JNEKFVLJN5FTXT46CVFQVGOA5BTDNL6TDB655V6WWMZWT5LC7FBQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:26.974479733 (89h49m26.845562768s) [noise=true , address=89.149.201.147:4023 , piece_id=LQPBNSGESUY2CONNWGJWPE67S76QATD3T2DICJPYJ3H4J2NRCWKQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:26.974507296 (89h49m26.845590341s) [seg_pos={0 0}, upload=78358]: Segment reached optimal threshold of 26 pieces. Dec 9 09:18:26.974751972 (89h49m26.845835017s) [noise=true , address=89.149.201.147:4007 , piece_id=PY2KA7ZSI422BIOOTFWMLFY3LVQIFS4DC2VWTBR5HUN5C36DR53Q, seg_pos={0 0}, upload=78358]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew): protocol: expected piece hash; context canceled; EOF Dec 9 09:18:26.974762383 (89h49m26.845845438s) [noise=true , address=89.149.201.147:4007 , piece_id=PY2KA7ZSI422BIOOTFWMLFY3LVQIFS4DC2VWTBR5HUN5C36DR53Q, seg_pos={0 0}, upload=78358]: Long tail cancellation due to optimal threshold Dec 9 09:18:26.974851171 (89h49m26.845934226s) [noise=true , address=89.149.201.147:4019 , piece_id=JI735NFBOCR6YSMRDKNIH653LHV3D5LGY34PJMPISBZE7TV57MCA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wgZQR7Ezekmvf9cNwCZ8CmZBjZgoqWC3N2UE3n4YQSmBsqQQ2): protocol: expected piece hash; context canceled; EOF Dec 9 09:18:26.974873604 (89h49m26.845956669s) [noise=true , address=89.149.201.147:4019 , piece_id=JI735NFBOCR6YSMRDKNIH653LHV3D5LGY34PJMPISBZE7TV57MCA, seg_pos={0 0}, upload=78358]: Long tail cancellation due to optimal threshold Dec 9 09:18:26.974923419 (89h49m26.846006464s) [seg_pos={0 0}, upload=78358]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 09:18:26.975251043 (89h49m26.846334138s) [upload=78358]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 09:18:26.975262835 (89h49m26.84634587s) [upload=78358]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 09:19:23.276531408 (89h50m23.147614453s) [upload=78358]: Done uploading segments... Dec 9 09:19:23.276661235 (89h50m23.14774428s) [upload=78358]: Upload finished. err: metaclient: context deadline exceeded Duration: 13.339365522s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.....................................X...X..............X...] duration: 5.693598673s trace: https://jaeger.dev.storj.io/trace/9ef0062b2b9b30b FAIL (2025-12-09 09:48:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.69362367s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...X......X.................................................] duration: 3.40199792s trace: https://jaeger.dev.storj.io/trace/62479de42d709d4f FAIL (2025-12-09 09:48:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.402017077s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 701.54651ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/754b20513c76ffd8 FAIL (2025-12-09 09:48:39 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 701.574905ms core satellite us1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 13.503291542s trace: https://jaeger.dev.storj.io/trace/2278184cf7cb09d3 FAIL (2025-12-09 09:48:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== HDOWUF7N2LWP3MU4DSDXO3YAI6KVCUP2US3A6OX6WIGW3XYQ, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.957067453 (89h49m23.828150528s) [noise=true , address=81.171.14.70:4031 , piece_id=TL65276T2XDWZUTNUZ7IBI5D4BSAMB5S6W6G5CT3NZ6JF2WSESOA, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.958192827 (89h49m23.829275862s) [noise=true , address=185.167.204.205:4015 , piece_id=YU77JIVCOQ37G52QR4RKZTQ2WY7ME4JPHBM7RTUHBHAM5A4DCFDQ, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.963154113 (89h49m23.834237159s) [noise=true , address=185.167.204.206:4065 , piece_id=LQIMSCI7CLY57EOLIERM5NSR64QSNR64FD6H2L6OUQV6VANBP77Q, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.964026705 (89h49m23.83510977s) [noise=true , address=185.167.204.206:4057 , piece_id=AVPMO43TLI7KCC52Y4OH2W2G3RLMGLBU7BTGPEO5FG2VUN7272DA, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.967189518 (89h49m23.838272563s) [noise=true , address=81.171.14.70:4030 , piece_id=6VDC74RCXTDZDGRMAGDR6HKRMEZAUCVM2KXIORPRFHOUPZ4SRURQ, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.969161493 (89h49m23.840244548s) [noise=true , address=89.149.201.147:4019 , piece_id=JI735NFBOCR6YSMRDKNIH653LHV3D5LGY34PJMPISBZE7TV57MCA, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.969242788 (89h49m23.840325834s) [noise=true , address=185.167.204.206:4060 , piece_id=TR5JMA65VL3X2JYKINUGVORZFKKBKV2NY2LBLHPQJS5Z4P2NJHWA, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.970553794 (89h49m23.841636849s) [noise=true , address=185.167.204.205:4016 , piece_id=T4SMB72E6JS76626B3SOUTKCFK6OHRPEQSKOLCBBQW5XY5PO3TNA, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.972813026 (89h49m23.843896081s) [noise=true , address=185.167.204.206:4054 , piece_id=CBV3DDAZUBWN4RVNVA37BC5ESANQPT4MYGHWMNTO4AZJ32ZZ7LWQ, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.978089925 (89h49m23.84917298s) [noise=true , address=185.167.204.205:4036 , piece_id=JE2RTZWLPINK4DZEIN5VZSF7NQNDIJ4EEEHLNXIGCXZAVVK23RGQ, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:24.785720741 (89h49m24.656803786s) [upload=78358]: Next returned nil segment Dec 9 09:18:24.785736831 (89h49m24.656819876s) [upload=78358]: Waiting for error group managing segments... Dec 9 09:18:25.168467403 (89h49m25.039550448s) [noise=true , address=185.167.204.206:4049 , piece_id=MEZGGAN2ZFVW535D3QU5FIF5X2XNI67T7FMNIJQMMXLHF5NN7KCA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.185302831 (89h49m25.056385876s) [noise=true , address=185.167.204.206:4054 , piece_id=CBV3DDAZUBWN4RVNVA37BC5ESANQPT4MYGHWMNTO4AZJ32ZZ7LWQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.185793916 (89h49m25.056876961s) [noise=true , address=185.167.204.206:4057 , piece_id=AVPMO43TLI7KCC52Y4OH2W2G3RLMGLBU7BTGPEO5FG2VUN7272DA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.186032270 (89h49m25.057115315s) [noise=true , address=185.167.204.206:4050 , piece_id=LBHTVVRMFZERIJPYLJGIZB6FC4YERLQI4YL2OMUXKN37OLQHDHDA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.188698217 (89h49m25.059781262s) [noise=true , address=185.167.204.205:4016 , piece_id=T4SMB72E6JS76626B3SOUTKCFK6OHRPEQSKOLCBBQW5XY5PO3TNA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.198062969 (89h49m25.069146014s) [noise=true , address=185.167.204.205:4024 , piece_id=Q53WJVADCT7IWGNY7OUXAA4VGI6ARFLTGDLM2KM6LBE2HBBA3DAQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.201189182 (89h49m25.072272227s) [noise=true , address=185.167.204.206:4060 , piece_id=TR5JMA65VL3X2JYKINUGVORZFKKBKV2NY2LBLHPQJS5Z4P2NJHWA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.202370381 (89h49m25.073453426s) [noise=true , address=185.167.204.206:4069 , piece_id=LD44HDOWUF7N2LWP3MU4DSDXO3YAI6KVCUP2US3A6OX6WIGW3XYQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.205089780 (89h49m25.076172816s) [noise=true , address=185.167.204.205:4032 , piece_id=R75QHC7MZRLI6HXHLCBWU5ZCKUQQI35I6URWVG7WKDIAFWIPN7XQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.207830660 (89h49m25.078913705s) [noise=true , address=185.167.204.206:4037 , piece_id=NW4QWZFM7JA6UF7JNUWNPEYL76BLWLB5KF3VLHGJZSCORTKL5SOA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.208925113 (89h49m25.080008159s) [noise=true , address=185.167.204.205:4036 , piece_id=JE2RTZWLPINK4DZEIN5VZSF7NQNDIJ4EEEHLNXIGCXZAVVK23RGQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.209472386 (89h49m25.080555431s) [noise=true , address=185.167.204.206:4065 , piece_id=LQIMSCI7CLY57EOLIERM5NSR64QSNR64FD6H2L6OUQV6VANBP77Q, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.211577764 (89h49m25.082660809s) [noise=true , address=185.167.204.205:4017 , piece_id=MILTVCWHWQJBMZFZPFXXCDSTPWRYHDHBFQEOPO6YG5PQTMD3RONQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.219514148 (89h49m25.090597183s) [noise=true , address=185.167.204.205:4015 , piece_id=YU77JIVCOQ37G52QR4RKZTQ2WY7ME4JPHBM7RTUHBHAM5A4DCFDQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.769337000 (89h49m25.640420045s) [noise=true , address=81.171.14.70:4039 , piece_id=QFA6JIAMTETMRGCCK6PJDUKWH43PZJOQQ4X5Y5ILCGL5R37OYTUQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.772473423 (89h49m25.643556468s) [noise=true , address=81.171.14.70:4032 , piece_id=FJAX2O76ZV2L5FWWXABF7FGT5JUQ6A3YDMKJUQSREBEJPIGQP7RA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.800364573 (89h49m25.671447608s) [noise=true , address=81.171.14.70:4030 , piece_id=6VDC74RCXTDZDGRMAGDR6HKRMEZAUCVM2KXIORPRFHOUPZ4SRURQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.800677198 (89h49m25.671760243s) [noise=true , address=81.171.14.70:4047 , piece_id=WZJLBCHCHZHOHL5IC66ZWGIF4EOHNOEH3XBVBXIZCV65YRSTNPBA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.810545961 (89h49m25.681629006s) [noise=true , address=81.171.14.70:4046 , piece_id=CZRVAU2PNZBUSVCYX5MOV72SZXID4GPJ7BDPPD4NIJ235EY62YMA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.838216351 (89h49m25.709299396s) [noise=true , address=81.171.14.70:4031 , piece_id=TL65276T2XDWZUTNUZ7IBI5D4BSAMB5S6W6G5CT3NZ6JF2WSESOA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:26.250350354 (89h49m26.121433399s) [noise=true , address=81.171.14.70:4036 , piece_id=3PRKMFHXBCM52IQ3IUGSQR5G727WYKUTVJBVC5QERPZ2WFCGSA3Q, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:26.277784646 (89h49m26.148867691s) [noise=true , address=81.171.14.70:4048 , piece_id=PTOL6CIPLQS6LTCCLI6PFEXOAOXHTMUQ4SYTRRJRWSXTED4VHMFQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:26.519380506 (89h49m26.390463552s) [noise=true , address=89.149.201.147:4005 , piece_id=HVHGMPYGLWAMDJVZTNUVXZ3GCPWDKDF3SBCJO2ILKKQGSPAYEBFA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:26.910352793 (89h49m26.781435828s) [noise=true , address=89.149.201.147:4001 , piece_id=PJ3CG6DO2HMSDREM23CEEBEDEEMREOBD2C2WMZSM2B5MSRASY6GQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:26.962433886 (89h49m26.833516921s) [noise=true , address=89.149.201.147:4004 , piece_id=JNEKFVLJN5FTXT46CVFQVGOA5BTDNL6TDB655V6WWMZWT5LC7FBQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:26.974479733 (89h49m26.845562768s) [noise=true , address=89.149.201.147:4023 , piece_id=LQPBNSGESUY2CONNWGJWPE67S76QATD3T2DICJPYJ3H4J2NRCWKQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:26.974507296 (89h49m26.845590341s) [seg_pos={0 0}, upload=78358]: Segment reached optimal threshold of 26 pieces. Dec 9 09:18:26.974751972 (89h49m26.845835017s) [noise=true , address=89.149.201.147:4007 , piece_id=PY2KA7ZSI422BIOOTFWMLFY3LVQIFS4DC2VWTBR5HUN5C36DR53Q, seg_pos={0 0}, upload=78358]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew): protocol: expected piece hash; context canceled; EOF Dec 9 09:18:26.974762383 (89h49m26.845845438s) [noise=true , address=89.149.201.147:4007 , piece_id=PY2KA7ZSI422BIOOTFWMLFY3LVQIFS4DC2VWTBR5HUN5C36DR53Q, seg_pos={0 0}, upload=78358]: Long tail cancellation due to optimal threshold Dec 9 09:18:26.974851171 (89h49m26.845934226s) [noise=true , address=89.149.201.147:4019 , piece_id=JI735NFBOCR6YSMRDKNIH653LHV3D5LGY34PJMPISBZE7TV57MCA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wgZQR7Ezekmvf9cNwCZ8CmZBjZgoqWC3N2UE3n4YQSmBsqQQ2): protocol: expected piece hash; context canceled; EOF Dec 9 09:18:26.974873604 (89h49m26.845956669s) [noise=true , address=89.149.201.147:4019 , piece_id=JI735NFBOCR6YSMRDKNIH653LHV3D5LGY34PJMPISBZE7TV57MCA, seg_pos={0 0}, upload=78358]: Long tail cancellation due to optimal threshold Dec 9 09:18:26.974923419 (89h49m26.846006464s) [seg_pos={0 0}, upload=78358]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 09:18:26.975251043 (89h49m26.846334138s) [upload=78358]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 09:18:26.975262835 (89h49m26.84634587s) [upload=78358]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 09:19:23.276531408 (89h50m23.147614453s) [upload=78358]: Done uploading segments... Dec 9 09:19:23.276661235 (89h50m23.14774428s) [upload=78358]: Upload finished. err: metaclient: context deadline exceeded Duration: 13.50332752s core uplink eu-select-1 eu1-placement-15
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 694.788481ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/754b20513c76ffd8 FAIL (2025-12-09 09:48:19 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 694.812756ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 1.977086344s trace: https://jaeger.dev.storj.io/trace/92f373d508d696c FAIL (2025-12-09 09:47:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 09:26:29.076085885 (89h57m28.94716892s) [upload=78482]: Starting upload Dec 9 09:26:29.076139197 (89h57m28.947222232s) [upload=78482]: Done starting upload Dec 9 09:26:29.076210933 (89h57m28.947293978s) [upload=78482]: Uploading segments... Dec 9 09:26:29.076295604 (89h57m28.947378639s) [upload=78482]: Got next segment. Inline: true Dec 9 09:26:29.076303269 (89h57m28.947386314s) [upload=78482]: Waiting for error group managing segments... Dec 9 09:26:29.076399502 (89h57m28.947482537s) [upload=78482]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 09:26:29.076427595 (89h57m28.94751064s) [upload=78482]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 9 09:26:29.076432515 (89h57m28.94751555s) [upload=78482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 09:27:28.400202146 (89h58m28.271285192s) [upload=78482]: Done uploading segments... Dec 9 09:27:28.400236482 (89h58m28.271319517s) [upload=78482]: Upload finished. err: metaclient: context deadline exceeded Duration:* 1.977118175s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [......................................X...X..............X..] duration: 5.586707703s trace: https://jaeger.dev.storj.io/trace/9ef0062b2b9b30b FAIL (2025-12-09 09:47:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.586740336s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 14.354712347s trace: https://jaeger.dev.storj.io/trace/2278184cf7cb09d3 FAIL (2025-12-09 09:47:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== HDOWUF7N2LWP3MU4DSDXO3YAI6KVCUP2US3A6OX6WIGW3XYQ, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.957067453 (89h49m23.828150528s) [noise=true , address=81.171.14.70:4031 , piece_id=TL65276T2XDWZUTNUZ7IBI5D4BSAMB5S6W6G5CT3NZ6JF2WSESOA, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.958192827 (89h49m23.829275862s) [noise=true , address=185.167.204.205:4015 , piece_id=YU77JIVCOQ37G52QR4RKZTQ2WY7ME4JPHBM7RTUHBHAM5A4DCFDQ, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.963154113 (89h49m23.834237159s) [noise=true , address=185.167.204.206:4065 , piece_id=LQIMSCI7CLY57EOLIERM5NSR64QSNR64FD6H2L6OUQV6VANBP77Q, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.964026705 (89h49m23.83510977s) [noise=true , address=185.167.204.206:4057 , piece_id=AVPMO43TLI7KCC52Y4OH2W2G3RLMGLBU7BTGPEO5FG2VUN7272DA, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.967189518 (89h49m23.838272563s) [noise=true , address=81.171.14.70:4030 , piece_id=6VDC74RCXTDZDGRMAGDR6HKRMEZAUCVM2KXIORPRFHOUPZ4SRURQ, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.969161493 (89h49m23.840244548s) [noise=true , address=89.149.201.147:4019 , piece_id=JI735NFBOCR6YSMRDKNIH653LHV3D5LGY34PJMPISBZE7TV57MCA, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.969242788 (89h49m23.840325834s) [noise=true , address=185.167.204.206:4060 , piece_id=TR5JMA65VL3X2JYKINUGVORZFKKBKV2NY2LBLHPQJS5Z4P2NJHWA, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.970553794 (89h49m23.841636849s) [noise=true , address=185.167.204.205:4016 , piece_id=T4SMB72E6JS76626B3SOUTKCFK6OHRPEQSKOLCBBQW5XY5PO3TNA, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.972813026 (89h49m23.843896081s) [noise=true , address=185.167.204.206:4054 , piece_id=CBV3DDAZUBWN4RVNVA37BC5ESANQPT4MYGHWMNTO4AZJ32ZZ7LWQ, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.978089925 (89h49m23.84917298s) [noise=true , address=185.167.204.205:4036 , piece_id=JE2RTZWLPINK4DZEIN5VZSF7NQNDIJ4EEEHLNXIGCXZAVVK23RGQ, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:24.785720741 (89h49m24.656803786s) [upload=78358]: Next returned nil segment Dec 9 09:18:24.785736831 (89h49m24.656819876s) [upload=78358]: Waiting for error group managing segments... Dec 9 09:18:25.168467403 (89h49m25.039550448s) [noise=true , address=185.167.204.206:4049 , piece_id=MEZGGAN2ZFVW535D3QU5FIF5X2XNI67T7FMNIJQMMXLHF5NN7KCA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.185302831 (89h49m25.056385876s) [noise=true , address=185.167.204.206:4054 , piece_id=CBV3DDAZUBWN4RVNVA37BC5ESANQPT4MYGHWMNTO4AZJ32ZZ7LWQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.185793916 (89h49m25.056876961s) [noise=true , address=185.167.204.206:4057 , piece_id=AVPMO43TLI7KCC52Y4OH2W2G3RLMGLBU7BTGPEO5FG2VUN7272DA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.186032270 (89h49m25.057115315s) [noise=true , address=185.167.204.206:4050 , piece_id=LBHTVVRMFZERIJPYLJGIZB6FC4YERLQI4YL2OMUXKN37OLQHDHDA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.188698217 (89h49m25.059781262s) [noise=true , address=185.167.204.205:4016 , piece_id=T4SMB72E6JS76626B3SOUTKCFK6OHRPEQSKOLCBBQW5XY5PO3TNA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.198062969 (89h49m25.069146014s) [noise=true , address=185.167.204.205:4024 , piece_id=Q53WJVADCT7IWGNY7OUXAA4VGI6ARFLTGDLM2KM6LBE2HBBA3DAQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.201189182 (89h49m25.072272227s) [noise=true , address=185.167.204.206:4060 , piece_id=TR5JMA65VL3X2JYKINUGVORZFKKBKV2NY2LBLHPQJS5Z4P2NJHWA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.202370381 (89h49m25.073453426s) [noise=true , address=185.167.204.206:4069 , piece_id=LD44HDOWUF7N2LWP3MU4DSDXO3YAI6KVCUP2US3A6OX6WIGW3XYQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.205089780 (89h49m25.076172816s) [noise=true , address=185.167.204.205:4032 , piece_id=R75QHC7MZRLI6HXHLCBWU5ZCKUQQI35I6URWVG7WKDIAFWIPN7XQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.207830660 (89h49m25.078913705s) [noise=true , address=185.167.204.206:4037 , piece_id=NW4QWZFM7JA6UF7JNUWNPEYL76BLWLB5KF3VLHGJZSCORTKL5SOA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.208925113 (89h49m25.080008159s) [noise=true , address=185.167.204.205:4036 , piece_id=JE2RTZWLPINK4DZEIN5VZSF7NQNDIJ4EEEHLNXIGCXZAVVK23RGQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.209472386 (89h49m25.080555431s) [noise=true , address=185.167.204.206:4065 , piece_id=LQIMSCI7CLY57EOLIERM5NSR64QSNR64FD6H2L6OUQV6VANBP77Q, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.211577764 (89h49m25.082660809s) [noise=true , address=185.167.204.205:4017 , piece_id=MILTVCWHWQJBMZFZPFXXCDSTPWRYHDHBFQEOPO6YG5PQTMD3RONQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.219514148 (89h49m25.090597183s) [noise=true , address=185.167.204.205:4015 , piece_id=YU77JIVCOQ37G52QR4RKZTQ2WY7ME4JPHBM7RTUHBHAM5A4DCFDQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.769337000 (89h49m25.640420045s) [noise=true , address=81.171.14.70:4039 , piece_id=QFA6JIAMTETMRGCCK6PJDUKWH43PZJOQQ4X5Y5ILCGL5R37OYTUQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.772473423 (89h49m25.643556468s) [noise=true , address=81.171.14.70:4032 , piece_id=FJAX2O76ZV2L5FWWXABF7FGT5JUQ6A3YDMKJUQSREBEJPIGQP7RA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.800364573 (89h49m25.671447608s) [noise=true , address=81.171.14.70:4030 , piece_id=6VDC74RCXTDZDGRMAGDR6HKRMEZAUCVM2KXIORPRFHOUPZ4SRURQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.800677198 (89h49m25.671760243s) [noise=true , address=81.171.14.70:4047 , piece_id=WZJLBCHCHZHOHL5IC66ZWGIF4EOHNOEH3XBVBXIZCV65YRSTNPBA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.810545961 (89h49m25.681629006s) [noise=true , address=81.171.14.70:4046 , piece_id=CZRVAU2PNZBUSVCYX5MOV72SZXID4GPJ7BDPPD4NIJ235EY62YMA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.838216351 (89h49m25.709299396s) [noise=true , address=81.171.14.70:4031 , piece_id=TL65276T2XDWZUTNUZ7IBI5D4BSAMB5S6W6G5CT3NZ6JF2WSESOA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:26.250350354 (89h49m26.121433399s) [noise=true , address=81.171.14.70:4036 , piece_id=3PRKMFHXBCM52IQ3IUGSQR5G727WYKUTVJBVC5QERPZ2WFCGSA3Q, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:26.277784646 (89h49m26.148867691s) [noise=true , address=81.171.14.70:4048 , piece_id=PTOL6CIPLQS6LTCCLI6PFEXOAOXHTMUQ4SYTRRJRWSXTED4VHMFQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:26.519380506 (89h49m26.390463552s) [noise=true , address=89.149.201.147:4005 , piece_id=HVHGMPYGLWAMDJVZTNUVXZ3GCPWDKDF3SBCJO2ILKKQGSPAYEBFA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:26.910352793 (89h49m26.781435828s) [noise=true , address=89.149.201.147:4001 , piece_id=PJ3CG6DO2HMSDREM23CEEBEDEEMREOBD2C2WMZSM2B5MSRASY6GQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:26.962433886 (89h49m26.833516921s) [noise=true , address=89.149.201.147:4004 , piece_id=JNEKFVLJN5FTXT46CVFQVGOA5BTDNL6TDB655V6WWMZWT5LC7FBQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:26.974479733 (89h49m26.845562768s) [noise=true , address=89.149.201.147:4023 , piece_id=LQPBNSGESUY2CONNWGJWPE67S76QATD3T2DICJPYJ3H4J2NRCWKQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:26.974507296 (89h49m26.845590341s) [seg_pos={0 0}, upload=78358]: Segment reached optimal threshold of 26 pieces. Dec 9 09:18:26.974751972 (89h49m26.845835017s) [noise=true , address=89.149.201.147:4007 , piece_id=PY2KA7ZSI422BIOOTFWMLFY3LVQIFS4DC2VWTBR5HUN5C36DR53Q, seg_pos={0 0}, upload=78358]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew): protocol: expected piece hash; context canceled; EOF Dec 9 09:18:26.974762383 (89h49m26.845845438s) [noise=true , address=89.149.201.147:4007 , piece_id=PY2KA7ZSI422BIOOTFWMLFY3LVQIFS4DC2VWTBR5HUN5C36DR53Q, seg_pos={0 0}, upload=78358]: Long tail cancellation due to optimal threshold Dec 9 09:18:26.974851171 (89h49m26.845934226s) [noise=true , address=89.149.201.147:4019 , piece_id=JI735NFBOCR6YSMRDKNIH653LHV3D5LGY34PJMPISBZE7TV57MCA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wgZQR7Ezekmvf9cNwCZ8CmZBjZgoqWC3N2UE3n4YQSmBsqQQ2): protocol: expected piece hash; context canceled; EOF Dec 9 09:18:26.974873604 (89h49m26.845956669s) [noise=true , address=89.149.201.147:4019 , piece_id=JI735NFBOCR6YSMRDKNIH653LHV3D5LGY34PJMPISBZE7TV57MCA, seg_pos={0 0}, upload=78358]: Long tail cancellation due to optimal threshold Dec 9 09:18:26.974923419 (89h49m26.846006464s) [seg_pos={0 0}, upload=78358]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 09:18:26.975251043 (89h49m26.846334138s) [upload=78358]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 09:18:26.975262835 (89h49m26.84634587s) [upload=78358]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 09:19:23.276531408 (89h50m23.147614453s) [upload=78358]: Done uploading segments... Dec 9 09:19:23.276661235 (89h50m23.14774428s) [upload=78358]: Upload finished. err: metaclient: context deadline exceeded Duration: 14.354738348s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [....X......X................................................] duration: 3.391283508s trace: https://jaeger.dev.storj.io/trace/62479de42d709d4f FAIL (2025-12-09 09:47:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.391304597s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 688.856606ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/754b20513c76ffd8 FAIL (2025-12-09 09:47:16 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 688.883236ms core satellite us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.......................................X...X..............X.] duration: 5.689235701s trace: https://jaeger.dev.storj.io/trace/9ef0062b2b9b30b FAIL (2025-12-09 09:47:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.689268382s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 1.974507929s trace: https://jaeger.dev.storj.io/trace/92f373d508d696c FAIL (2025-12-09 09:46:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 09:26:29.076085885 (89h57m28.94716892s) [upload=78482]: Starting upload Dec 9 09:26:29.076139197 (89h57m28.947222232s) [upload=78482]: Done starting upload Dec 9 09:26:29.076210933 (89h57m28.947293978s) [upload=78482]: Uploading segments... Dec 9 09:26:29.076295604 (89h57m28.947378639s) [upload=78482]: Got next segment. Inline: true Dec 9 09:26:29.076303269 (89h57m28.947386314s) [upload=78482]: Waiting for error group managing segments... Dec 9 09:26:29.076399502 (89h57m28.947482537s) [upload=78482]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 09:26:29.076427595 (89h57m28.94751064s) [upload=78482]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 9 09:26:29.076432515 (89h57m28.94751555s) [upload=78482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 09:27:28.400202146 (89h58m28.271285192s) [upload=78482]: Done uploading segments... Dec 9 09:27:28.400236482 (89h58m28.271319517s) [upload=78482]: Upload finished. err: metaclient: context deadline exceeded Duration:* 1.974533549s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [........................................X...X..............X] duration: 1m0.000513496s trace: https://jaeger.dev.storj.io/trace/9ef0062b2b9b30b FAIL (2025-12-09 09:46:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000535338s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 709.600286ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/754b20513c76ffd8 FAIL (2025-12-09 09:46:53 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 709.629361ms core satellite us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.....X......X...............................................] duration: 3.457052526s trace: https://jaeger.dev.storj.io/trace/62479de42d709d4f FAIL (2025-12-09 09:46:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.457109725s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 4.623924743s trace: https://jaeger.dev.storj.io/trace/2278184cf7cb09d3 FAIL (2025-12-09 09:46:28 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== HDOWUF7N2LWP3MU4DSDXO3YAI6KVCUP2US3A6OX6WIGW3XYQ, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.957067453 (89h49m23.828150528s) [noise=true , address=81.171.14.70:4031 , piece_id=TL65276T2XDWZUTNUZ7IBI5D4BSAMB5S6W6G5CT3NZ6JF2WSESOA, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.958192827 (89h49m23.829275862s) [noise=true , address=185.167.204.205:4015 , piece_id=YU77JIVCOQ37G52QR4RKZTQ2WY7ME4JPHBM7RTUHBHAM5A4DCFDQ, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.963154113 (89h49m23.834237159s) [noise=true , address=185.167.204.206:4065 , piece_id=LQIMSCI7CLY57EOLIERM5NSR64QSNR64FD6H2L6OUQV6VANBP77Q, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.964026705 (89h49m23.83510977s) [noise=true , address=185.167.204.206:4057 , piece_id=AVPMO43TLI7KCC52Y4OH2W2G3RLMGLBU7BTGPEO5FG2VUN7272DA, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.967189518 (89h49m23.838272563s) [noise=true , address=81.171.14.70:4030 , piece_id=6VDC74RCXTDZDGRMAGDR6HKRMEZAUCVM2KXIORPRFHOUPZ4SRURQ, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.969161493 (89h49m23.840244548s) [noise=true , address=89.149.201.147:4019 , piece_id=JI735NFBOCR6YSMRDKNIH653LHV3D5LGY34PJMPISBZE7TV57MCA, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.969242788 (89h49m23.840325834s) [noise=true , address=185.167.204.206:4060 , piece_id=TR5JMA65VL3X2JYKINUGVORZFKKBKV2NY2LBLHPQJS5Z4P2NJHWA, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.970553794 (89h49m23.841636849s) [noise=true , address=185.167.204.205:4016 , piece_id=T4SMB72E6JS76626B3SOUTKCFK6OHRPEQSKOLCBBQW5XY5PO3TNA, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.972813026 (89h49m23.843896081s) [noise=true , address=185.167.204.206:4054 , piece_id=CBV3DDAZUBWN4RVNVA37BC5ESANQPT4MYGHWMNTO4AZJ32ZZ7LWQ, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.978089925 (89h49m23.84917298s) [noise=true , address=185.167.204.205:4036 , piece_id=JE2RTZWLPINK4DZEIN5VZSF7NQNDIJ4EEEHLNXIGCXZAVVK23RGQ, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:24.785720741 (89h49m24.656803786s) [upload=78358]: Next returned nil segment Dec 9 09:18:24.785736831 (89h49m24.656819876s) [upload=78358]: Waiting for error group managing segments... Dec 9 09:18:25.168467403 (89h49m25.039550448s) [noise=true , address=185.167.204.206:4049 , piece_id=MEZGGAN2ZFVW535D3QU5FIF5X2XNI67T7FMNIJQMMXLHF5NN7KCA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.185302831 (89h49m25.056385876s) [noise=true , address=185.167.204.206:4054 , piece_id=CBV3DDAZUBWN4RVNVA37BC5ESANQPT4MYGHWMNTO4AZJ32ZZ7LWQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.185793916 (89h49m25.056876961s) [noise=true , address=185.167.204.206:4057 , piece_id=AVPMO43TLI7KCC52Y4OH2W2G3RLMGLBU7BTGPEO5FG2VUN7272DA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.186032270 (89h49m25.057115315s) [noise=true , address=185.167.204.206:4050 , piece_id=LBHTVVRMFZERIJPYLJGIZB6FC4YERLQI4YL2OMUXKN37OLQHDHDA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.188698217 (89h49m25.059781262s) [noise=true , address=185.167.204.205:4016 , piece_id=T4SMB72E6JS76626B3SOUTKCFK6OHRPEQSKOLCBBQW5XY5PO3TNA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.198062969 (89h49m25.069146014s) [noise=true , address=185.167.204.205:4024 , piece_id=Q53WJVADCT7IWGNY7OUXAA4VGI6ARFLTGDLM2KM6LBE2HBBA3DAQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.201189182 (89h49m25.072272227s) [noise=true , address=185.167.204.206:4060 , piece_id=TR5JMA65VL3X2JYKINUGVORZFKKBKV2NY2LBLHPQJS5Z4P2NJHWA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.202370381 (89h49m25.073453426s) [noise=true , address=185.167.204.206:4069 , piece_id=LD44HDOWUF7N2LWP3MU4DSDXO3YAI6KVCUP2US3A6OX6WIGW3XYQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.205089780 (89h49m25.076172816s) [noise=true , address=185.167.204.205:4032 , piece_id=R75QHC7MZRLI6HXHLCBWU5ZCKUQQI35I6URWVG7WKDIAFWIPN7XQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.207830660 (89h49m25.078913705s) [noise=true , address=185.167.204.206:4037 , piece_id=NW4QWZFM7JA6UF7JNUWNPEYL76BLWLB5KF3VLHGJZSCORTKL5SOA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.208925113 (89h49m25.080008159s) [noise=true , address=185.167.204.205:4036 , piece_id=JE2RTZWLPINK4DZEIN5VZSF7NQNDIJ4EEEHLNXIGCXZAVVK23RGQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.209472386 (89h49m25.080555431s) [noise=true , address=185.167.204.206:4065 , piece_id=LQIMSCI7CLY57EOLIERM5NSR64QSNR64FD6H2L6OUQV6VANBP77Q, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.211577764 (89h49m25.082660809s) [noise=true , address=185.167.204.205:4017 , piece_id=MILTVCWHWQJBMZFZPFXXCDSTPWRYHDHBFQEOPO6YG5PQTMD3RONQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.219514148 (89h49m25.090597183s) [noise=true , address=185.167.204.205:4015 , piece_id=YU77JIVCOQ37G52QR4RKZTQ2WY7ME4JPHBM7RTUHBHAM5A4DCFDQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.769337000 (89h49m25.640420045s) [noise=true , address=81.171.14.70:4039 , piece_id=QFA6JIAMTETMRGCCK6PJDUKWH43PZJOQQ4X5Y5ILCGL5R37OYTUQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.772473423 (89h49m25.643556468s) [noise=true , address=81.171.14.70:4032 , piece_id=FJAX2O76ZV2L5FWWXABF7FGT5JUQ6A3YDMKJUQSREBEJPIGQP7RA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.800364573 (89h49m25.671447608s) [noise=true , address=81.171.14.70:4030 , piece_id=6VDC74RCXTDZDGRMAGDR6HKRMEZAUCVM2KXIORPRFHOUPZ4SRURQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.800677198 (89h49m25.671760243s) [noise=true , address=81.171.14.70:4047 , piece_id=WZJLBCHCHZHOHL5IC66ZWGIF4EOHNOEH3XBVBXIZCV65YRSTNPBA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.810545961 (89h49m25.681629006s) [noise=true , address=81.171.14.70:4046 , piece_id=CZRVAU2PNZBUSVCYX5MOV72SZXID4GPJ7BDPPD4NIJ235EY62YMA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.838216351 (89h49m25.709299396s) [noise=true , address=81.171.14.70:4031 , piece_id=TL65276T2XDWZUTNUZ7IBI5D4BSAMB5S6W6G5CT3NZ6JF2WSESOA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:26.250350354 (89h49m26.121433399s) [noise=true , address=81.171.14.70:4036 , piece_id=3PRKMFHXBCM52IQ3IUGSQR5G727WYKUTVJBVC5QERPZ2WFCGSA3Q, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:26.277784646 (89h49m26.148867691s) [noise=true , address=81.171.14.70:4048 , piece_id=PTOL6CIPLQS6LTCCLI6PFEXOAOXHTMUQ4SYTRRJRWSXTED4VHMFQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:26.519380506 (89h49m26.390463552s) [noise=true , address=89.149.201.147:4005 , piece_id=HVHGMPYGLWAMDJVZTNUVXZ3GCPWDKDF3SBCJO2ILKKQGSPAYEBFA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:26.910352793 (89h49m26.781435828s) [noise=true , address=89.149.201.147:4001 , piece_id=PJ3CG6DO2HMSDREM23CEEBEDEEMREOBD2C2WMZSM2B5MSRASY6GQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:26.962433886 (89h49m26.833516921s) [noise=true , address=89.149.201.147:4004 , piece_id=JNEKFVLJN5FTXT46CVFQVGOA5BTDNL6TDB655V6WWMZWT5LC7FBQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:26.974479733 (89h49m26.845562768s) [noise=true , address=89.149.201.147:4023 , piece_id=LQPBNSGESUY2CONNWGJWPE67S76QATD3T2DICJPYJ3H4J2NRCWKQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:26.974507296 (89h49m26.845590341s) [seg_pos={0 0}, upload=78358]: Segment reached optimal threshold of 26 pieces. Dec 9 09:18:26.974751972 (89h49m26.845835017s) [noise=true , address=89.149.201.147:4007 , piece_id=PY2KA7ZSI422BIOOTFWMLFY3LVQIFS4DC2VWTBR5HUN5C36DR53Q, seg_pos={0 0}, upload=78358]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew): protocol: expected piece hash; context canceled; EOF Dec 9 09:18:26.974762383 (89h49m26.845845438s) [noise=true , address=89.149.201.147:4007 , piece_id=PY2KA7ZSI422BIOOTFWMLFY3LVQIFS4DC2VWTBR5HUN5C36DR53Q, seg_pos={0 0}, upload=78358]: Long tail cancellation due to optimal threshold Dec 9 09:18:26.974851171 (89h49m26.845934226s) [noise=true , address=89.149.201.147:4019 , piece_id=JI735NFBOCR6YSMRDKNIH653LHV3D5LGY34PJMPISBZE7TV57MCA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wgZQR7Ezekmvf9cNwCZ8CmZBjZgoqWC3N2UE3n4YQSmBsqQQ2): protocol: expected piece hash; context canceled; EOF Dec 9 09:18:26.974873604 (89h49m26.845956669s) [noise=true , address=89.149.201.147:4019 , piece_id=JI735NFBOCR6YSMRDKNIH653LHV3D5LGY34PJMPISBZE7TV57MCA, seg_pos={0 0}, upload=78358]: Long tail cancellation due to optimal threshold Dec 9 09:18:26.974923419 (89h49m26.846006464s) [seg_pos={0 0}, upload=78358]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 09:18:26.975251043 (89h49m26.846334138s) [upload=78358]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 09:18:26.975262835 (89h49m26.84634587s) [upload=78358]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 09:19:23.276531408 (89h50m23.147614453s) [upload=78358]: Done uploading segments... Dec 9 09:19:23.276661235 (89h50m23.14774428s) [upload=78358]: Upload finished. err: metaclient: context deadline exceeded Duration: 4.623953829s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [X.....X......X..............................................] duration: 3.526053717s trace: https://jaeger.dev.storj.io/trace/62479de42d709d4f FAIL (2025-12-09 09:46:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.526099574s core uplink us-select-1 us1-placement-20
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 686.009586ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/754b20513c76ffd8 FAIL (2025-12-09 09:45:49 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 686.037819ms core satellite us1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.X.....X......X.............................................] duration: 3.435968157s trace: https://jaeger.dev.storj.io/trace/62479de42d709d4f FAIL (2025-12-09 09:45:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.435994377s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 1.988727241s trace: https://jaeger.dev.storj.io/trace/92f373d508d696c FAIL (2025-12-09 09:45:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 09:26:29.076085885 (89h57m28.94716892s) [upload=78482]: Starting upload Dec 9 09:26:29.076139197 (89h57m28.947222232s) [upload=78482]: Done starting upload Dec 9 09:26:29.076210933 (89h57m28.947293978s) [upload=78482]: Uploading segments... Dec 9 09:26:29.076295604 (89h57m28.947378639s) [upload=78482]: Got next segment. Inline: true Dec 9 09:26:29.076303269 (89h57m28.947386314s) [upload=78482]: Waiting for error group managing segments... Dec 9 09:26:29.076399502 (89h57m28.947482537s) [upload=78482]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 09:26:29.076427595 (89h57m28.94751064s) [upload=78482]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 9 09:26:29.076432515 (89h57m28.94751555s) [upload=78482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 09:27:28.400202146 (89h58m28.271285192s) [upload=78482]: Done uploading segments... Dec 9 09:27:28.400236482 (89h58m28.271319517s) [upload=78482]: Upload finished. err: metaclient: context deadline exceeded Duration:* 1.988747718s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 9.91868702s trace: https://jaeger.dev.storj.io/trace/2278184cf7cb09d3 FAIL (2025-12-09 09:45:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== HDOWUF7N2LWP3MU4DSDXO3YAI6KVCUP2US3A6OX6WIGW3XYQ, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.957067453 (89h49m23.828150528s) [noise=true , address=81.171.14.70:4031 , piece_id=TL65276T2XDWZUTNUZ7IBI5D4BSAMB5S6W6G5CT3NZ6JF2WSESOA, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.958192827 (89h49m23.829275862s) [noise=true , address=185.167.204.205:4015 , piece_id=YU77JIVCOQ37G52QR4RKZTQ2WY7ME4JPHBM7RTUHBHAM5A4DCFDQ, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.963154113 (89h49m23.834237159s) [noise=true , address=185.167.204.206:4065 , piece_id=LQIMSCI7CLY57EOLIERM5NSR64QSNR64FD6H2L6OUQV6VANBP77Q, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.964026705 (89h49m23.83510977s) [noise=true , address=185.167.204.206:4057 , piece_id=AVPMO43TLI7KCC52Y4OH2W2G3RLMGLBU7BTGPEO5FG2VUN7272DA, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.967189518 (89h49m23.838272563s) [noise=true , address=81.171.14.70:4030 , piece_id=6VDC74RCXTDZDGRMAGDR6HKRMEZAUCVM2KXIORPRFHOUPZ4SRURQ, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.969161493 (89h49m23.840244548s) [noise=true , address=89.149.201.147:4019 , piece_id=JI735NFBOCR6YSMRDKNIH653LHV3D5LGY34PJMPISBZE7TV57MCA, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.969242788 (89h49m23.840325834s) [noise=true , address=185.167.204.206:4060 , piece_id=TR5JMA65VL3X2JYKINUGVORZFKKBKV2NY2LBLHPQJS5Z4P2NJHWA, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.970553794 (89h49m23.841636849s) [noise=true , address=185.167.204.205:4016 , piece_id=T4SMB72E6JS76626B3SOUTKCFK6OHRPEQSKOLCBBQW5XY5PO3TNA, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.972813026 (89h49m23.843896081s) [noise=true , address=185.167.204.206:4054 , piece_id=CBV3DDAZUBWN4RVNVA37BC5ESANQPT4MYGHWMNTO4AZJ32ZZ7LWQ, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.978089925 (89h49m23.84917298s) [noise=true , address=185.167.204.205:4036 , piece_id=JE2RTZWLPINK4DZEIN5VZSF7NQNDIJ4EEEHLNXIGCXZAVVK23RGQ, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:24.785720741 (89h49m24.656803786s) [upload=78358]: Next returned nil segment Dec 9 09:18:24.785736831 (89h49m24.656819876s) [upload=78358]: Waiting for error group managing segments... Dec 9 09:18:25.168467403 (89h49m25.039550448s) [noise=true , address=185.167.204.206:4049 , piece_id=MEZGGAN2ZFVW535D3QU5FIF5X2XNI67T7FMNIJQMMXLHF5NN7KCA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.185302831 (89h49m25.056385876s) [noise=true , address=185.167.204.206:4054 , piece_id=CBV3DDAZUBWN4RVNVA37BC5ESANQPT4MYGHWMNTO4AZJ32ZZ7LWQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.185793916 (89h49m25.056876961s) [noise=true , address=185.167.204.206:4057 , piece_id=AVPMO43TLI7KCC52Y4OH2W2G3RLMGLBU7BTGPEO5FG2VUN7272DA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.186032270 (89h49m25.057115315s) [noise=true , address=185.167.204.206:4050 , piece_id=LBHTVVRMFZERIJPYLJGIZB6FC4YERLQI4YL2OMUXKN37OLQHDHDA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.188698217 (89h49m25.059781262s) [noise=true , address=185.167.204.205:4016 , piece_id=T4SMB72E6JS76626B3SOUTKCFK6OHRPEQSKOLCBBQW5XY5PO3TNA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.198062969 (89h49m25.069146014s) [noise=true , address=185.167.204.205:4024 , piece_id=Q53WJVADCT7IWGNY7OUXAA4VGI6ARFLTGDLM2KM6LBE2HBBA3DAQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.201189182 (89h49m25.072272227s) [noise=true , address=185.167.204.206:4060 , piece_id=TR5JMA65VL3X2JYKINUGVORZFKKBKV2NY2LBLHPQJS5Z4P2NJHWA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.202370381 (89h49m25.073453426s) [noise=true , address=185.167.204.206:4069 , piece_id=LD44HDOWUF7N2LWP3MU4DSDXO3YAI6KVCUP2US3A6OX6WIGW3XYQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.205089780 (89h49m25.076172816s) [noise=true , address=185.167.204.205:4032 , piece_id=R75QHC7MZRLI6HXHLCBWU5ZCKUQQI35I6URWVG7WKDIAFWIPN7XQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.207830660 (89h49m25.078913705s) [noise=true , address=185.167.204.206:4037 , piece_id=NW4QWZFM7JA6UF7JNUWNPEYL76BLWLB5KF3VLHGJZSCORTKL5SOA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.208925113 (89h49m25.080008159s) [noise=true , address=185.167.204.205:4036 , piece_id=JE2RTZWLPINK4DZEIN5VZSF7NQNDIJ4EEEHLNXIGCXZAVVK23RGQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.209472386 (89h49m25.080555431s) [noise=true , address=185.167.204.206:4065 , piece_id=LQIMSCI7CLY57EOLIERM5NSR64QSNR64FD6H2L6OUQV6VANBP77Q, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.211577764 (89h49m25.082660809s) [noise=true , address=185.167.204.205:4017 , piece_id=MILTVCWHWQJBMZFZPFXXCDSTPWRYHDHBFQEOPO6YG5PQTMD3RONQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.219514148 (89h49m25.090597183s) [noise=true , address=185.167.204.205:4015 , piece_id=YU77JIVCOQ37G52QR4RKZTQ2WY7ME4JPHBM7RTUHBHAM5A4DCFDQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.769337000 (89h49m25.640420045s) [noise=true , address=81.171.14.70:4039 , piece_id=QFA6JIAMTETMRGCCK6PJDUKWH43PZJOQQ4X5Y5ILCGL5R37OYTUQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.772473423 (89h49m25.643556468s) [noise=true , address=81.171.14.70:4032 , piece_id=FJAX2O76ZV2L5FWWXABF7FGT5JUQ6A3YDMKJUQSREBEJPIGQP7RA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.800364573 (89h49m25.671447608s) [noise=true , address=81.171.14.70:4030 , piece_id=6VDC74RCXTDZDGRMAGDR6HKRMEZAUCVM2KXIORPRFHOUPZ4SRURQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.800677198 (89h49m25.671760243s) [noise=true , address=81.171.14.70:4047 , piece_id=WZJLBCHCHZHOHL5IC66ZWGIF4EOHNOEH3XBVBXIZCV65YRSTNPBA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.810545961 (89h49m25.681629006s) [noise=true , address=81.171.14.70:4046 , piece_id=CZRVAU2PNZBUSVCYX5MOV72SZXID4GPJ7BDPPD4NIJ235EY62YMA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.838216351 (89h49m25.709299396s) [noise=true , address=81.171.14.70:4031 , piece_id=TL65276T2XDWZUTNUZ7IBI5D4BSAMB5S6W6G5CT3NZ6JF2WSESOA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:26.250350354 (89h49m26.121433399s) [noise=true , address=81.171.14.70:4036 , piece_id=3PRKMFHXBCM52IQ3IUGSQR5G727WYKUTVJBVC5QERPZ2WFCGSA3Q, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:26.277784646 (89h49m26.148867691s) [noise=true , address=81.171.14.70:4048 , piece_id=PTOL6CIPLQS6LTCCLI6PFEXOAOXHTMUQ4SYTRRJRWSXTED4VHMFQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:26.519380506 (89h49m26.390463552s) [noise=true , address=89.149.201.147:4005 , piece_id=HVHGMPYGLWAMDJVZTNUVXZ3GCPWDKDF3SBCJO2ILKKQGSPAYEBFA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:26.910352793 (89h49m26.781435828s) [noise=true , address=89.149.201.147:4001 , piece_id=PJ3CG6DO2HMSDREM23CEEBEDEEMREOBD2C2WMZSM2B5MSRASY6GQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:26.962433886 (89h49m26.833516921s) [noise=true , address=89.149.201.147:4004 , piece_id=JNEKFVLJN5FTXT46CVFQVGOA5BTDNL6TDB655V6WWMZWT5LC7FBQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:26.974479733 (89h49m26.845562768s) [noise=true , address=89.149.201.147:4023 , piece_id=LQPBNSGESUY2CONNWGJWPE67S76QATD3T2DICJPYJ3H4J2NRCWKQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:26.974507296 (89h49m26.845590341s) [seg_pos={0 0}, upload=78358]: Segment reached optimal threshold of 26 pieces. Dec 9 09:18:26.974751972 (89h49m26.845835017s) [noise=true , address=89.149.201.147:4007 , piece_id=PY2KA7ZSI422BIOOTFWMLFY3LVQIFS4DC2VWTBR5HUN5C36DR53Q, seg_pos={0 0}, upload=78358]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew): protocol: expected piece hash; context canceled; EOF Dec 9 09:18:26.974762383 (89h49m26.845845438s) [noise=true , address=89.149.201.147:4007 , piece_id=PY2KA7ZSI422BIOOTFWMLFY3LVQIFS4DC2VWTBR5HUN5C36DR53Q, seg_pos={0 0}, upload=78358]: Long tail cancellation due to optimal threshold Dec 9 09:18:26.974851171 (89h49m26.845934226s) [noise=true , address=89.149.201.147:4019 , piece_id=JI735NFBOCR6YSMRDKNIH653LHV3D5LGY34PJMPISBZE7TV57MCA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wgZQR7Ezekmvf9cNwCZ8CmZBjZgoqWC3N2UE3n4YQSmBsqQQ2): protocol: expected piece hash; context canceled; EOF Dec 9 09:18:26.974873604 (89h49m26.845956669s) [noise=true , address=89.149.201.147:4019 , piece_id=JI735NFBOCR6YSMRDKNIH653LHV3D5LGY34PJMPISBZE7TV57MCA, seg_pos={0 0}, upload=78358]: Long tail cancellation due to optimal threshold Dec 9 09:18:26.974923419 (89h49m26.846006464s) [seg_pos={0 0}, upload=78358]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 09:18:26.975251043 (89h49m26.846334138s) [upload=78358]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 09:18:26.975262835 (89h49m26.84634587s) [upload=78358]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 09:19:23.276531408 (89h50m23.147614453s) [upload=78358]: Done uploading segments... Dec 9 09:19:23.276661235 (89h50m23.14774428s) [upload=78358]: Upload finished. err: metaclient: context deadline exceeded Duration: 9.918717527s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..X.....X......X............................................] duration: 3.449518871s trace: https://jaeger.dev.storj.io/trace/62479de42d709d4f FAIL (2025-12-09 09:45:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.449550209s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [.........................................X...X..............] duration: 6.041007757s trace: https://jaeger.dev.storj.io/trace/2d73c2387989a005 FAIL (2025-12-09 09:44:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.041052522s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 4.624466933s trace: https://jaeger.dev.storj.io/trace/2278184cf7cb09d3 FAIL (2025-12-09 09:44:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== HDOWUF7N2LWP3MU4DSDXO3YAI6KVCUP2US3A6OX6WIGW3XYQ, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.957067453 (89h49m23.828150528s) [noise=true , address=81.171.14.70:4031 , piece_id=TL65276T2XDWZUTNUZ7IBI5D4BSAMB5S6W6G5CT3NZ6JF2WSESOA, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.958192827 (89h49m23.829275862s) [noise=true , address=185.167.204.205:4015 , piece_id=YU77JIVCOQ37G52QR4RKZTQ2WY7ME4JPHBM7RTUHBHAM5A4DCFDQ, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.963154113 (89h49m23.834237159s) [noise=true , address=185.167.204.206:4065 , piece_id=LQIMSCI7CLY57EOLIERM5NSR64QSNR64FD6H2L6OUQV6VANBP77Q, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.964026705 (89h49m23.83510977s) [noise=true , address=185.167.204.206:4057 , piece_id=AVPMO43TLI7KCC52Y4OH2W2G3RLMGLBU7BTGPEO5FG2VUN7272DA, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.967189518 (89h49m23.838272563s) [noise=true , address=81.171.14.70:4030 , piece_id=6VDC74RCXTDZDGRMAGDR6HKRMEZAUCVM2KXIORPRFHOUPZ4SRURQ, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.969161493 (89h49m23.840244548s) [noise=true , address=89.149.201.147:4019 , piece_id=JI735NFBOCR6YSMRDKNIH653LHV3D5LGY34PJMPISBZE7TV57MCA, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.969242788 (89h49m23.840325834s) [noise=true , address=185.167.204.206:4060 , piece_id=TR5JMA65VL3X2JYKINUGVORZFKKBKV2NY2LBLHPQJS5Z4P2NJHWA, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.970553794 (89h49m23.841636849s) [noise=true , address=185.167.204.205:4016 , piece_id=T4SMB72E6JS76626B3SOUTKCFK6OHRPEQSKOLCBBQW5XY5PO3TNA, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.972813026 (89h49m23.843896081s) [noise=true , address=185.167.204.206:4054 , piece_id=CBV3DDAZUBWN4RVNVA37BC5ESANQPT4MYGHWMNTO4AZJ32ZZ7LWQ, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.978089925 (89h49m23.84917298s) [noise=true , address=185.167.204.205:4036 , piece_id=JE2RTZWLPINK4DZEIN5VZSF7NQNDIJ4EEEHLNXIGCXZAVVK23RGQ, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:24.785720741 (89h49m24.656803786s) [upload=78358]: Next returned nil segment Dec 9 09:18:24.785736831 (89h49m24.656819876s) [upload=78358]: Waiting for error group managing segments... Dec 9 09:18:25.168467403 (89h49m25.039550448s) [noise=true , address=185.167.204.206:4049 , piece_id=MEZGGAN2ZFVW535D3QU5FIF5X2XNI67T7FMNIJQMMXLHF5NN7KCA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.185302831 (89h49m25.056385876s) [noise=true , address=185.167.204.206:4054 , piece_id=CBV3DDAZUBWN4RVNVA37BC5ESANQPT4MYGHWMNTO4AZJ32ZZ7LWQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.185793916 (89h49m25.056876961s) [noise=true , address=185.167.204.206:4057 , piece_id=AVPMO43TLI7KCC52Y4OH2W2G3RLMGLBU7BTGPEO5FG2VUN7272DA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.186032270 (89h49m25.057115315s) [noise=true , address=185.167.204.206:4050 , piece_id=LBHTVVRMFZERIJPYLJGIZB6FC4YERLQI4YL2OMUXKN37OLQHDHDA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.188698217 (89h49m25.059781262s) [noise=true , address=185.167.204.205:4016 , piece_id=T4SMB72E6JS76626B3SOUTKCFK6OHRPEQSKOLCBBQW5XY5PO3TNA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.198062969 (89h49m25.069146014s) [noise=true , address=185.167.204.205:4024 , piece_id=Q53WJVADCT7IWGNY7OUXAA4VGI6ARFLTGDLM2KM6LBE2HBBA3DAQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.201189182 (89h49m25.072272227s) [noise=true , address=185.167.204.206:4060 , piece_id=TR5JMA65VL3X2JYKINUGVORZFKKBKV2NY2LBLHPQJS5Z4P2NJHWA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.202370381 (89h49m25.073453426s) [noise=true , address=185.167.204.206:4069 , piece_id=LD44HDOWUF7N2LWP3MU4DSDXO3YAI6KVCUP2US3A6OX6WIGW3XYQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.205089780 (89h49m25.076172816s) [noise=true , address=185.167.204.205:4032 , piece_id=R75QHC7MZRLI6HXHLCBWU5ZCKUQQI35I6URWVG7WKDIAFWIPN7XQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.207830660 (89h49m25.078913705s) [noise=true , address=185.167.204.206:4037 , piece_id=NW4QWZFM7JA6UF7JNUWNPEYL76BLWLB5KF3VLHGJZSCORTKL5SOA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.208925113 (89h49m25.080008159s) [noise=true , address=185.167.204.205:4036 , piece_id=JE2RTZWLPINK4DZEIN5VZSF7NQNDIJ4EEEHLNXIGCXZAVVK23RGQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.209472386 (89h49m25.080555431s) [noise=true , address=185.167.204.206:4065 , piece_id=LQIMSCI7CLY57EOLIERM5NSR64QSNR64FD6H2L6OUQV6VANBP77Q, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.211577764 (89h49m25.082660809s) [noise=true , address=185.167.204.205:4017 , piece_id=MILTVCWHWQJBMZFZPFXXCDSTPWRYHDHBFQEOPO6YG5PQTMD3RONQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.219514148 (89h49m25.090597183s) [noise=true , address=185.167.204.205:4015 , piece_id=YU77JIVCOQ37G52QR4RKZTQ2WY7ME4JPHBM7RTUHBHAM5A4DCFDQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.769337000 (89h49m25.640420045s) [noise=true , address=81.171.14.70:4039 , piece_id=QFA6JIAMTETMRGCCK6PJDUKWH43PZJOQQ4X5Y5ILCGL5R37OYTUQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.772473423 (89h49m25.643556468s) [noise=true , address=81.171.14.70:4032 , piece_id=FJAX2O76ZV2L5FWWXABF7FGT5JUQ6A3YDMKJUQSREBEJPIGQP7RA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.800364573 (89h49m25.671447608s) [noise=true , address=81.171.14.70:4030 , piece_id=6VDC74RCXTDZDGRMAGDR6HKRMEZAUCVM2KXIORPRFHOUPZ4SRURQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.800677198 (89h49m25.671760243s) [noise=true , address=81.171.14.70:4047 , piece_id=WZJLBCHCHZHOHL5IC66ZWGIF4EOHNOEH3XBVBXIZCV65YRSTNPBA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.810545961 (89h49m25.681629006s) [noise=true , address=81.171.14.70:4046 , piece_id=CZRVAU2PNZBUSVCYX5MOV72SZXID4GPJ7BDPPD4NIJ235EY62YMA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.838216351 (89h49m25.709299396s) [noise=true , address=81.171.14.70:4031 , piece_id=TL65276T2XDWZUTNUZ7IBI5D4BSAMB5S6W6G5CT3NZ6JF2WSESOA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:26.250350354 (89h49m26.121433399s) [noise=true , address=81.171.14.70:4036 , piece_id=3PRKMFHXBCM52IQ3IUGSQR5G727WYKUTVJBVC5QERPZ2WFCGSA3Q, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:26.277784646 (89h49m26.148867691s) [noise=true , address=81.171.14.70:4048 , piece_id=PTOL6CIPLQS6LTCCLI6PFEXOAOXHTMUQ4SYTRRJRWSXTED4VHMFQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:26.519380506 (89h49m26.390463552s) [noise=true , address=89.149.201.147:4005 , piece_id=HVHGMPYGLWAMDJVZTNUVXZ3GCPWDKDF3SBCJO2ILKKQGSPAYEBFA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:26.910352793 (89h49m26.781435828s) [noise=true , address=89.149.201.147:4001 , piece_id=PJ3CG6DO2HMSDREM23CEEBEDEEMREOBD2C2WMZSM2B5MSRASY6GQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:26.962433886 (89h49m26.833516921s) [noise=true , address=89.149.201.147:4004 , piece_id=JNEKFVLJN5FTXT46CVFQVGOA5BTDNL6TDB655V6WWMZWT5LC7FBQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:26.974479733 (89h49m26.845562768s) [noise=true , address=89.149.201.147:4023 , piece_id=LQPBNSGESUY2CONNWGJWPE67S76QATD3T2DICJPYJ3H4J2NRCWKQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:26.974507296 (89h49m26.845590341s) [seg_pos={0 0}, upload=78358]: Segment reached optimal threshold of 26 pieces. Dec 9 09:18:26.974751972 (89h49m26.845835017s) [noise=true , address=89.149.201.147:4007 , piece_id=PY2KA7ZSI422BIOOTFWMLFY3LVQIFS4DC2VWTBR5HUN5C36DR53Q, seg_pos={0 0}, upload=78358]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew): protocol: expected piece hash; context canceled; EOF Dec 9 09:18:26.974762383 (89h49m26.845845438s) [noise=true , address=89.149.201.147:4007 , piece_id=PY2KA7ZSI422BIOOTFWMLFY3LVQIFS4DC2VWTBR5HUN5C36DR53Q, seg_pos={0 0}, upload=78358]: Long tail cancellation due to optimal threshold Dec 9 09:18:26.974851171 (89h49m26.845934226s) [noise=true , address=89.149.201.147:4019 , piece_id=JI735NFBOCR6YSMRDKNIH653LHV3D5LGY34PJMPISBZE7TV57MCA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wgZQR7Ezekmvf9cNwCZ8CmZBjZgoqWC3N2UE3n4YQSmBsqQQ2): protocol: expected piece hash; context canceled; EOF Dec 9 09:18:26.974873604 (89h49m26.845956669s) [noise=true , address=89.149.201.147:4019 , piece_id=JI735NFBOCR6YSMRDKNIH653LHV3D5LGY34PJMPISBZE7TV57MCA, seg_pos={0 0}, upload=78358]: Long tail cancellation due to optimal threshold Dec 9 09:18:26.974923419 (89h49m26.846006464s) [seg_pos={0 0}, upload=78358]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 09:18:26.975251043 (89h49m26.846334138s) [upload=78358]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 09:18:26.975262835 (89h49m26.84634587s) [upload=78358]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 09:19:23.276531408 (89h50m23.147614453s) [upload=78358]: Done uploading segments... Dec 9 09:19:23.276661235 (89h50m23.14774428s) [upload=78358]: Upload finished. err: metaclient: context deadline exceeded Duration: 4.624496039s core uplink eu-select-1 eu1-placement-15
checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 684.692887ms remote: "tcp:35.212.10.183:443" target: https://us1.storj.io/login trace: https://jaeger.dev.storj.io/trace/754b20513c76ffd8 FAIL (2025-12-09 09:44:47 UTC) orbiter warning: Get "https://us1.storj.io/login": EOF Duration: 684.720148ms core satellite us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 1.910887888s trace: https://jaeger.dev.storj.io/trace/92f373d508d696c FAIL (2025-12-09 09:44:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 09:26:29.076085885 (89h57m28.94716892s) [upload=78482]: Starting upload Dec 9 09:26:29.076139197 (89h57m28.947222232s) [upload=78482]: Done starting upload Dec 9 09:26:29.076210933 (89h57m28.947293978s) [upload=78482]: Uploading segments... Dec 9 09:26:29.076295604 (89h57m28.947378639s) [upload=78482]: Got next segment. Inline: true Dec 9 09:26:29.076303269 (89h57m28.947386314s) [upload=78482]: Waiting for error group managing segments... Dec 9 09:26:29.076399502 (89h57m28.947482537s) [upload=78482]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 09:26:29.076427595 (89h57m28.94751064s) [upload=78482]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 9 09:26:29.076432515 (89h57m28.94751555s) [upload=78482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 09:27:28.400202146 (89h58m28.271285192s) [upload=78482]: Done uploading segments... Dec 9 09:27:28.400236482 (89h58m28.271319517s) [upload=78482]: Upload finished. err: metaclient: context deadline exceeded Duration:* 1.910915721s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...X.....X......X...........................................] duration: 3.598030576s trace: https://jaeger.dev.storj.io/trace/62479de42d709d4f FAIL (2025-12-09 09:44:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.598056995s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 1.966962894s trace: https://jaeger.dev.storj.io/trace/92f373d508d696c FAIL (2025-12-09 09:44:03 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 9 09:26:29.076085885 (89h57m28.94716892s) [upload=78482]: Starting upload Dec 9 09:26:29.076139197 (89h57m28.947222232s) [upload=78482]: Done starting upload Dec 9 09:26:29.076210933 (89h57m28.947293978s) [upload=78482]: Uploading segments... Dec 9 09:26:29.076295604 (89h57m28.947378639s) [upload=78482]: Got next segment. Inline: true Dec 9 09:26:29.076303269 (89h57m28.947386314s) [upload=78482]: Waiting for error group managing segments... Dec 9 09:26:29.076399502 (89h57m28.947482537s) [upload=78482]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 9 09:26:29.076427595 (89h57m28.94751064s) [upload=78482]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 9 09:26:29.076432515 (89h57m28.94751555s) [upload=78482]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 09:27:28.400202146 (89h58m28.271285192s) [upload=78482]: Done uploading segments... Dec 9 09:27:28.400236482 (89h58m28.271319517s) [upload=78482]: Upload finished. err: metaclient: context deadline exceeded Duration:* 1.966994344s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..........................................X...X.............] duration: 5.611242409s trace: https://jaeger.dev.storj.io/trace/2d73c2387989a005 FAIL (2025-12-09 09:44:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.611277455s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 5.359020015s trace: https://jaeger.dev.storj.io/trace/2278184cf7cb09d3 FAIL (2025-12-09 09:43:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== HDOWUF7N2LWP3MU4DSDXO3YAI6KVCUP2US3A6OX6WIGW3XYQ, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.957067453 (89h49m23.828150528s) [noise=true , address=81.171.14.70:4031 , piece_id=TL65276T2XDWZUTNUZ7IBI5D4BSAMB5S6W6G5CT3NZ6JF2WSESOA, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.958192827 (89h49m23.829275862s) [noise=true , address=185.167.204.205:4015 , piece_id=YU77JIVCOQ37G52QR4RKZTQ2WY7ME4JPHBM7RTUHBHAM5A4DCFDQ, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.963154113 (89h49m23.834237159s) [noise=true , address=185.167.204.206:4065 , piece_id=LQIMSCI7CLY57EOLIERM5NSR64QSNR64FD6H2L6OUQV6VANBP77Q, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.964026705 (89h49m23.83510977s) [noise=true , address=185.167.204.206:4057 , piece_id=AVPMO43TLI7KCC52Y4OH2W2G3RLMGLBU7BTGPEO5FG2VUN7272DA, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.967189518 (89h49m23.838272563s) [noise=true , address=81.171.14.70:4030 , piece_id=6VDC74RCXTDZDGRMAGDR6HKRMEZAUCVM2KXIORPRFHOUPZ4SRURQ, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.969161493 (89h49m23.840244548s) [noise=true , address=89.149.201.147:4019 , piece_id=JI735NFBOCR6YSMRDKNIH653LHV3D5LGY34PJMPISBZE7TV57MCA, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.969242788 (89h49m23.840325834s) [noise=true , address=185.167.204.206:4060 , piece_id=TR5JMA65VL3X2JYKINUGVORZFKKBKV2NY2LBLHPQJS5Z4P2NJHWA, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.970553794 (89h49m23.841636849s) [noise=true , address=185.167.204.205:4016 , piece_id=T4SMB72E6JS76626B3SOUTKCFK6OHRPEQSKOLCBBQW5XY5PO3TNA, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.972813026 (89h49m23.843896081s) [noise=true , address=185.167.204.206:4054 , piece_id=CBV3DDAZUBWN4RVNVA37BC5ESANQPT4MYGHWMNTO4AZJ32ZZ7LWQ, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:23.978089925 (89h49m23.84917298s) [noise=true , address=185.167.204.205:4036 , piece_id=JE2RTZWLPINK4DZEIN5VZSF7NQNDIJ4EEEHLNXIGCXZAVVK23RGQ, seg_pos={0 0}, upload=78358]: Uploading piece... Dec 9 09:18:24.785720741 (89h49m24.656803786s) [upload=78358]: Next returned nil segment Dec 9 09:18:24.785736831 (89h49m24.656819876s) [upload=78358]: Waiting for error group managing segments... Dec 9 09:18:25.168467403 (89h49m25.039550448s) [noise=true , address=185.167.204.206:4049 , piece_id=MEZGGAN2ZFVW535D3QU5FIF5X2XNI67T7FMNIJQMMXLHF5NN7KCA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.185302831 (89h49m25.056385876s) [noise=true , address=185.167.204.206:4054 , piece_id=CBV3DDAZUBWN4RVNVA37BC5ESANQPT4MYGHWMNTO4AZJ32ZZ7LWQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.185793916 (89h49m25.056876961s) [noise=true , address=185.167.204.206:4057 , piece_id=AVPMO43TLI7KCC52Y4OH2W2G3RLMGLBU7BTGPEO5FG2VUN7272DA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.186032270 (89h49m25.057115315s) [noise=true , address=185.167.204.206:4050 , piece_id=LBHTVVRMFZERIJPYLJGIZB6FC4YERLQI4YL2OMUXKN37OLQHDHDA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.188698217 (89h49m25.059781262s) [noise=true , address=185.167.204.205:4016 , piece_id=T4SMB72E6JS76626B3SOUTKCFK6OHRPEQSKOLCBBQW5XY5PO3TNA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.198062969 (89h49m25.069146014s) [noise=true , address=185.167.204.205:4024 , piece_id=Q53WJVADCT7IWGNY7OUXAA4VGI6ARFLTGDLM2KM6LBE2HBBA3DAQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.201189182 (89h49m25.072272227s) [noise=true , address=185.167.204.206:4060 , piece_id=TR5JMA65VL3X2JYKINUGVORZFKKBKV2NY2LBLHPQJS5Z4P2NJHWA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.202370381 (89h49m25.073453426s) [noise=true , address=185.167.204.206:4069 , piece_id=LD44HDOWUF7N2LWP3MU4DSDXO3YAI6KVCUP2US3A6OX6WIGW3XYQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.205089780 (89h49m25.076172816s) [noise=true , address=185.167.204.205:4032 , piece_id=R75QHC7MZRLI6HXHLCBWU5ZCKUQQI35I6URWVG7WKDIAFWIPN7XQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.207830660 (89h49m25.078913705s) [noise=true , address=185.167.204.206:4037 , piece_id=NW4QWZFM7JA6UF7JNUWNPEYL76BLWLB5KF3VLHGJZSCORTKL5SOA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.208925113 (89h49m25.080008159s) [noise=true , address=185.167.204.205:4036 , piece_id=JE2RTZWLPINK4DZEIN5VZSF7NQNDIJ4EEEHLNXIGCXZAVVK23RGQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.209472386 (89h49m25.080555431s) [noise=true , address=185.167.204.206:4065 , piece_id=LQIMSCI7CLY57EOLIERM5NSR64QSNR64FD6H2L6OUQV6VANBP77Q, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.211577764 (89h49m25.082660809s) [noise=true , address=185.167.204.205:4017 , piece_id=MILTVCWHWQJBMZFZPFXXCDSTPWRYHDHBFQEOPO6YG5PQTMD3RONQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.219514148 (89h49m25.090597183s) [noise=true , address=185.167.204.205:4015 , piece_id=YU77JIVCOQ37G52QR4RKZTQ2WY7ME4JPHBM7RTUHBHAM5A4DCFDQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.769337000 (89h49m25.640420045s) [noise=true , address=81.171.14.70:4039 , piece_id=QFA6JIAMTETMRGCCK6PJDUKWH43PZJOQQ4X5Y5ILCGL5R37OYTUQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.772473423 (89h49m25.643556468s) [noise=true , address=81.171.14.70:4032 , piece_id=FJAX2O76ZV2L5FWWXABF7FGT5JUQ6A3YDMKJUQSREBEJPIGQP7RA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.800364573 (89h49m25.671447608s) [noise=true , address=81.171.14.70:4030 , piece_id=6VDC74RCXTDZDGRMAGDR6HKRMEZAUCVM2KXIORPRFHOUPZ4SRURQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.800677198 (89h49m25.671760243s) [noise=true , address=81.171.14.70:4047 , piece_id=WZJLBCHCHZHOHL5IC66ZWGIF4EOHNOEH3XBVBXIZCV65YRSTNPBA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.810545961 (89h49m25.681629006s) [noise=true , address=81.171.14.70:4046 , piece_id=CZRVAU2PNZBUSVCYX5MOV72SZXID4GPJ7BDPPD4NIJ235EY62YMA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:25.838216351 (89h49m25.709299396s) [noise=true , address=81.171.14.70:4031 , piece_id=TL65276T2XDWZUTNUZ7IBI5D4BSAMB5S6W6G5CT3NZ6JF2WSESOA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:26.250350354 (89h49m26.121433399s) [noise=true , address=81.171.14.70:4036 , piece_id=3PRKMFHXBCM52IQ3IUGSQR5G727WYKUTVJBVC5QERPZ2WFCGSA3Q, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:26.277784646 (89h49m26.148867691s) [noise=true , address=81.171.14.70:4048 , piece_id=PTOL6CIPLQS6LTCCLI6PFEXOAOXHTMUQ4SYTRRJRWSXTED4VHMFQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:26.519380506 (89h49m26.390463552s) [noise=true , address=89.149.201.147:4005 , piece_id=HVHGMPYGLWAMDJVZTNUVXZ3GCPWDKDF3SBCJO2ILKKQGSPAYEBFA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:26.910352793 (89h49m26.781435828s) [noise=true , address=89.149.201.147:4001 , piece_id=PJ3CG6DO2HMSDREM23CEEBEDEEMREOBD2C2WMZSM2B5MSRASY6GQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:26.962433886 (89h49m26.833516921s) [noise=true , address=89.149.201.147:4004 , piece_id=JNEKFVLJN5FTXT46CVFQVGOA5BTDNL6TDB655V6WWMZWT5LC7FBQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:26.974479733 (89h49m26.845562768s) [noise=true , address=89.149.201.147:4023 , piece_id=LQPBNSGESUY2CONNWGJWPE67S76QATD3T2DICJPYJ3H4J2NRCWKQ, seg_pos={0 0}, upload=78358]: Done uploading piece. err: <nil> Dec 9 09:18:26.974507296 (89h49m26.845590341s) [seg_pos={0 0}, upload=78358]: Segment reached optimal threshold of 26 pieces. Dec 9 09:18:26.974751972 (89h49m26.845835017s) [noise=true , address=89.149.201.147:4007 , piece_id=PY2KA7ZSI422BIOOTFWMLFY3LVQIFS4DC2VWTBR5HUN5C36DR53Q, seg_pos={0 0}, upload=78358]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Q7d4vj1KoWhDr6xd9gEfFrZ1sV3Yro2AFke2f3Ke1rXsGnvew): protocol: expected piece hash; context canceled; EOF Dec 9 09:18:26.974762383 (89h49m26.845845438s) [noise=true , address=89.149.201.147:4007 , piece_id=PY2KA7ZSI422BIOOTFWMLFY3LVQIFS4DC2VWTBR5HUN5C36DR53Q, seg_pos={0 0}, upload=78358]: Long tail cancellation due to optimal threshold Dec 9 09:18:26.974851171 (89h49m26.845934226s) [noise=true , address=89.149.201.147:4019 , piece_id=JI735NFBOCR6YSMRDKNIH653LHV3D5LGY34PJMPISBZE7TV57MCA, seg_pos={0 0}, upload=78358]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12wgZQR7Ezekmvf9cNwCZ8CmZBjZgoqWC3N2UE3n4YQSmBsqQQ2): protocol: expected piece hash; context canceled; EOF Dec 9 09:18:26.974873604 (89h49m26.845956669s) [noise=true , address=89.149.201.147:4019 , piece_id=JI735NFBOCR6YSMRDKNIH653LHV3D5LGY34PJMPISBZE7TV57MCA, seg_pos={0 0}, upload=78358]: Long tail cancellation due to optimal threshold Dec 9 09:18:26.974923419 (89h49m26.846006464s) [seg_pos={0 0}, upload=78358]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil> Dec 9 09:18:26.975251043 (89h49m26.846334138s) [upload=78358]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 9 09:18:26.975262835 (89h49m26.84634587s) [upload=78358]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 9 09:19:23.276531408 (89h50m23.147614453s) [upload=78358]: Done uploading segments... Dec 9 09:19:23.276661235 (89h50m23.14774428s) [upload=78358]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.359048679s core uplink eu-select-1 eu1-placement-15

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)

checkers.HTTP{"GET" "https://us1.storj.io/login", TLSCertExpirationWarning: 672h0m0s, TLSCertExpirationError: 336h0m0s} failure (antiblip window 3/60): (http.us1.satellite_login)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1k", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.placement_20.inline)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)