Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 6.651624221s trace: https://jaeger.dev.storj.io/trace/4226536f67e7860 FAIL (2026-01-13 10:03:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.652052637s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 59 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.000589819s trace: https://jaeger.dev.storj.io/trace/6f238c8a9658684a FAIL (2026-01-13 10:03:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000612823s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 5.849945168s trace: https://jaeger.dev.storj.io/trace/4226536f67e7860 FAIL (2026-01-13 10:03:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.849958514s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 5.799098276s trace: https://jaeger.dev.storj.io/trace/209f6f5543c7dce4 FAIL (2026-01-13 10:03:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: NOBWZTUK4VDKXM3PCO3Y623KIW2XKFRZXWZEDFVVRT5JHTPCZZQQ) context deadline exceeded; piecestore: (Node ID: 12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj, Piece ID: 3FOIHL6K6JAEDPWIRHT7IJOQ534UUKYAVCNAW4V3R5QL3D25UKAA) context deadline exceeded; piecestore: (Node ID: 1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, Piece ID: KUG2JKJTNJWMTZAP5F4HGA2UEKRHORRYVFKQ2FAX4KLQDVEX7UYQ) context deadline exceeded; piecestore: (Node ID: 12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, Piece ID: LTQ6DD2KQ55UCWIJ43SWJO7Q5IDDJFTSD2Z7SVMHR2HLUOUSBNDQ) context deadline exceeded ======LOGS====== ent. Inline: false Jan 13 09:28:06.315478350 (929h59m6.186561385s) [upload=813972]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 13 09:28:06.315483540 (929h59m6.186566575s) [upload=813972]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 13 09:28:09.241621500 (929h59m9.112704545s) [seg_pos={0 0}, upload=813972]: Begin upload segment... Jan 13 09:28:09.241671425 (929h59m9.11275446s) [seg_pos={0 0}, upload=813972]: Done begin upload segment. Jan 13 09:28:09.241703947 (929h59m9.112786992s) [noise=true , address=145.79.227.47:4267 , piece_id=RB3RDZAJRF3VV6LL6LUI5544BM7MFTGUIQ64Y4G23FHFRIYWZNUA, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.241712664 (929h59m9.112795699s) [noise=true , address=145.79.227.49:4337 , piece_id=ZRIX2M4OOA6LKJ6FJ2JORHE4XZQDJRG3KSBUNRJ3EEEXMIK5VMYA, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.242438585 (929h59m9.11352162s) [noise=true , address=145.79.227.2:4113 , piece_id=KUG2JKJTNJWMTZAP5F4HGA2UEKRHORRYVFKQ2FAX4KLQDVEX7UYQ, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.242727556 (929h59m9.113810591s) [noise=true , address=145.79.227.35:4483 , piece_id=PH7UUSSVGW3I2KXRBDWEIFHDL3SC63SYDUPWMKG3NX3HKG6E3OYA, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.243047555 (929h59m9.1141306s) [noise=true , address=145.79.227.5:4221 , piece_id=MAYWAFPWGGY6OJMP2OPGXJ4CFWMHIOL32IDBYGFVIONN6PFRDK2Q, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.243340904 (929h59m9.114423939s) [noise=true , address=145.79.227.48:4301 , piece_id=BIBJBBTITB6AA2SWXXZAIEOOAIARHUIVCBXHB4JXMUSELMZSLLTA, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.243735925 (929h59m9.11481897s) [noise=true , address=145.79.227.37:4566 , piece_id=NHE62LRK3ZLPFLQXOIPJZ3LB47RSNU7L2C3DGJKZYV3EXLYBSN6Q, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.244030276 (929h59m9.115113312s) [noise=true , address=145.79.227.11:4732 , piece_id=ZNFPDXJOGVKRQHPN66JHECSZLNEQ57MVWX4GAA6ETQSCNWCJQ75Q, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.245863306 (929h59m9.116946341s) [noise=true , address=145.79.227.7:4293 , piece_id=LTQ6DD2KQ55UCWIJ43SWJO7Q5IDDJFTSD2Z7SVMHR2HLUOUSBNDQ, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.247027153 (929h59m9.118110188s) [noise=true , address=145.79.227.6:4257 , piece_id=7NPEYE7VHKUHSOYCDOUVMN2HBL4JEXAGZS64K7M6L6DDIIF755TQ, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.248440564 (929h59m9.119523619s) [noise=true , address=145.79.227.31:4799 , piece_id=NOBWZTUK4VDKXM3PCO3Y623KIW2XKFRZXWZEDFVVRT5JHTPCZZQQ, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.249844076 (929h59m9.120927121s) [noise=true , address=145.79.227.50:4373 , piece_id=R5A7RCLHAH3ODMP563M27O6T7ZNACL36HJJOFVLRHYG456XUXXBA, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.251275041 (929h59m9.122358076s) [noise=true , address=145.79.227.52:4447 , piece_id=ZZMECYYJKEXTVZI6WQVEMT23OAVZ7SILIHSSE2OHFKSESUAQHJRQ, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.252655870 (929h59m9.123738915s) [noise=true , address=145.79.227.21:4473 , piece_id=FHOGGYMZY7DU6T7FUFNALWHI4Y6SDMPYF242QLW7C4UZIKKXEJ3A, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.253473537 (929h59m9.124556582s) [noise=true , address=145.79.227.13:4803 , piece_id=DNNAZPZ2TYQI6VCXNIZKSMU5VQCXKEDN3L6K6MHSVNRN3R246BCQ, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.254758193 (929h59m9.125841238s) [noise=true , address=145.79.227.51:4409 , piece_id=3FOIHL6K6JAEDPWIRHT7IJOQ534UUKYAVCNAW4V3R5QL3D25UKAA, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:11.087868620 (929h59m10.958951665s) [upload=813972]: Next returned nil segment Jan 13 09:28:11.087884400 (929h59m10.958967435s) [upload=813972]: Waiting for error group managing segments... Jan 13 09:28:11.388729734 (929h59m11.259812799s) [noise=true , address=145.79.227.50:4373 , piece_id=R5A7RCLHAH3ODMP563M27O6T7ZNACL36HJJOFVLRHYG456XUXXBA, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.399388468 (929h59m11.270471504s) [noise=true , address=145.79.227.21:4473 , piece_id=FHOGGYMZY7DU6T7FUFNALWHI4Y6SDMPYF242QLW7C4UZIKKXEJ3A, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.405180828 (929h59m11.276263873s) [noise=true , address=145.79.227.13:4803 , piece_id=DNNAZPZ2TYQI6VCXNIZKSMU5VQCXKEDN3L6K6MHSVNRN3R246BCQ, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.423915039 (929h59m11.294998074s) [noise=true , address=145.79.227.31:4799 , piece_id=NOBWZTUK4VDKXM3PCO3Y623KIW2XKFRZXWZEDFVVRT5JHTPCZZQQ, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.425205065 (929h59m11.29628811s) [noise=true , address=145.79.227.48:4301 , piece_id=BIBJBBTITB6AA2SWXXZAIEOOAIARHUIVCBXHB4JXMUSELMZSLLTA, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.427897582 (929h59m11.298980617s) [noise=true , address=145.79.227.5:4221 , piece_id=MAYWAFPWGGY6OJMP2OPGXJ4CFWMHIOL32IDBYGFVIONN6PFRDK2Q, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.461421214 (929h59m11.332504249s) [noise=true , address=145.79.227.2:4113 , piece_id=KUG2JKJTNJWMTZAP5F4HGA2UEKRHORRYVFKQ2FAX4KLQDVEX7UYQ, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.466681119 (929h59m11.337764154s) [noise=true , address=145.79.227.37:4566 , piece_id=NHE62LRK3ZLPFLQXOIPJZ3LB47RSNU7L2C3DGJKZYV3EXLYBSN6Q, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.468074292 (929h59m11.339157327s) [noise=true , address=145.79.227.11:4732 , piece_id=ZNFPDXJOGVKRQHPN66JHECSZLNEQ57MVWX4GAA6ETQSCNWCJQ75Q, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.472717633 (929h59m11.343800668s) [noise=true , address=145.79.227.35:4483 , piece_id=PH7UUSSVGW3I2KXRBDWEIFHDL3SC63SYDUPWMKG3NX3HKG6E3OYA, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.502968303 (929h59m11.374051338s) [noise=true , address=145.79.227.7:4293 , piece_id=LTQ6DD2KQ55UCWIJ43SWJO7Q5IDDJFTSD2Z7SVMHR2HLUOUSBNDQ, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.554022174 (929h59m11.425105219s) [noise=true , address=145.79.227.51:4409 , piece_id=3FOIHL6K6JAEDPWIRHT7IJOQ534UUKYAVCNAW4V3R5QL3D25UKAA, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.585092324 (929h59m11.456175369s) [noise=true , address=145.79.227.47:4267 , piece_id=RB3RDZAJRF3VV6LL6LUI5544BM7MFTGUIQ64Y4G23FHFRIYWZNUA, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.585114155 (929h59m11.4561972s) [seg_pos={0 0}, upload=813972]: Segment reached optimal threshold of 13 pieces. Jan 13 09:28:11.585293828 (929h59m11.456376863s) [noise=true , address=145.79.227.52:4447 , piece_id=ZZMECYYJKEXTVZI6WQVEMT23OAVZ7SILIHSSE2OHFKSESUAQHJRQ, seg_pos={0 0}, upload=813972]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP): protocol: write tcp 188.166.225.170:44548->145.79.227.52:4447: use of closed network connection; write tcp 188.166.225.170:44548->145.79.227.52:4447: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44548->145.79.227.52:4447: use of closed network connection Jan 13 09:28:11.585299989 (929h59m11.456383025s) [noise=true , address=145.79.227.52:4447 , piece_id=ZZMECYYJKEXTVZI6WQVEMT23OAVZ7SILIHSSE2OHFKSESUAQHJRQ, seg_pos={0 0}, upload=813972]: Long tail cancellation due to optimal threshold Jan 13 09:28:11.585433102 (929h59m11.456516137s) [noise=true , address=145.79.227.6:4257 , piece_id=7NPEYE7VHKUHSOYCDOUVMN2HBL4JEXAGZS64K7M6L6DDIIF755TQ, seg_pos={0 0}, upload=813972]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK): protocol: write tcp 188.166.225.170:52450->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:52450->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52450->145.79.227.6:4257: use of closed network connection Jan 13 09:28:11.585440497 (929h59m11.456523532s) [noise=true , address=145.79.227.6:4257 , piece_id=7NPEYE7VHKUHSOYCDOUVMN2HBL4JEXAGZS64K7M6L6DDIIF755TQ, seg_pos={0 0}, upload=813972]: Long tail cancellation due to optimal threshold Jan 13 09:28:11.585617424 (929h59m11.456700469s) [noise=true , address=145.79.227.49:4337 , piece_id=ZRIX2M4OOA6LKJ6FJ2JORHE4XZQDJRG3KSBUNRJ3EEEXMIK5VMYA, seg_pos={0 0}, upload=813972]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: read tcp 188.166.225.170:49630->145.79.227.49:4337: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 13 09:28:11.585629516 (929h59m11.456712562s) [noise=true , address=145.79.227.49:4337 , piece_id=ZRIX2M4OOA6LKJ6FJ2JORHE4XZQDJRG3KSBUNRJ3EEEXMIK5VMYA, seg_pos={0 0}, upload=813972]: Long tail cancellation due to optimal threshold Jan 13 09:28:11.585663561 (929h59m11.456746596s) [seg_pos={0 0}, upload=813972]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 13 09:28:11.585783911 (929h59m11.456866946s) [upload=813972]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 13 09:28:11.585791646 (929h59m11.456874681s) [upload=813972]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 13 09:28:11.889773771 (929h59m11.760856816s) [upload=813972]: Done uploading segments... Jan 13 09:28:11.889820049 (929h59m11.760903094s) [upload=813972]: Upload finished. err: <nil> Duration: 5.799129975s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 59 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001381635s trace: https://jaeger.dev.storj.io/trace/63da1dae6fac5bd4 FAIL (2026-01-13 10:02:45 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001404879s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..............X....................X.......X..X....XX.......] duration: 28.576357747s trace: https://jaeger.dev.storj.io/trace/1422bf86419b0b28 FAIL (2026-01-13 10:02:42 UTC) orbiter warning: context deadline exceeded Duration: 28.576381983s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 7.689894082s trace: https://jaeger.dev.storj.io/trace/4226536f67e7860 FAIL (2026-01-13 10:02:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.689923967s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 5.815934749s trace: https://jaeger.dev.storj.io/trace/209f6f5543c7dce4 FAIL (2026-01-13 10:02:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: NOBWZTUK4VDKXM3PCO3Y623KIW2XKFRZXWZEDFVVRT5JHTPCZZQQ) context deadline exceeded; piecestore: (Node ID: 12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj, Piece ID: 3FOIHL6K6JAEDPWIRHT7IJOQ534UUKYAVCNAW4V3R5QL3D25UKAA) context deadline exceeded; piecestore: (Node ID: 1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, Piece ID: KUG2JKJTNJWMTZAP5F4HGA2UEKRHORRYVFKQ2FAX4KLQDVEX7UYQ) context deadline exceeded; piecestore: (Node ID: 12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, Piece ID: LTQ6DD2KQ55UCWIJ43SWJO7Q5IDDJFTSD2Z7SVMHR2HLUOUSBNDQ) context deadline exceeded ======LOGS====== ent. Inline: false Jan 13 09:28:06.315478350 (929h59m6.186561385s) [upload=813972]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 13 09:28:06.315483540 (929h59m6.186566575s) [upload=813972]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 13 09:28:09.241621500 (929h59m9.112704545s) [seg_pos={0 0}, upload=813972]: Begin upload segment... Jan 13 09:28:09.241671425 (929h59m9.11275446s) [seg_pos={0 0}, upload=813972]: Done begin upload segment. Jan 13 09:28:09.241703947 (929h59m9.112786992s) [noise=true , address=145.79.227.47:4267 , piece_id=RB3RDZAJRF3VV6LL6LUI5544BM7MFTGUIQ64Y4G23FHFRIYWZNUA, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.241712664 (929h59m9.112795699s) [noise=true , address=145.79.227.49:4337 , piece_id=ZRIX2M4OOA6LKJ6FJ2JORHE4XZQDJRG3KSBUNRJ3EEEXMIK5VMYA, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.242438585 (929h59m9.11352162s) [noise=true , address=145.79.227.2:4113 , piece_id=KUG2JKJTNJWMTZAP5F4HGA2UEKRHORRYVFKQ2FAX4KLQDVEX7UYQ, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.242727556 (929h59m9.113810591s) [noise=true , address=145.79.227.35:4483 , piece_id=PH7UUSSVGW3I2KXRBDWEIFHDL3SC63SYDUPWMKG3NX3HKG6E3OYA, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.243047555 (929h59m9.1141306s) [noise=true , address=145.79.227.5:4221 , piece_id=MAYWAFPWGGY6OJMP2OPGXJ4CFWMHIOL32IDBYGFVIONN6PFRDK2Q, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.243340904 (929h59m9.114423939s) [noise=true , address=145.79.227.48:4301 , piece_id=BIBJBBTITB6AA2SWXXZAIEOOAIARHUIVCBXHB4JXMUSELMZSLLTA, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.243735925 (929h59m9.11481897s) [noise=true , address=145.79.227.37:4566 , piece_id=NHE62LRK3ZLPFLQXOIPJZ3LB47RSNU7L2C3DGJKZYV3EXLYBSN6Q, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.244030276 (929h59m9.115113312s) [noise=true , address=145.79.227.11:4732 , piece_id=ZNFPDXJOGVKRQHPN66JHECSZLNEQ57MVWX4GAA6ETQSCNWCJQ75Q, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.245863306 (929h59m9.116946341s) [noise=true , address=145.79.227.7:4293 , piece_id=LTQ6DD2KQ55UCWIJ43SWJO7Q5IDDJFTSD2Z7SVMHR2HLUOUSBNDQ, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.247027153 (929h59m9.118110188s) [noise=true , address=145.79.227.6:4257 , piece_id=7NPEYE7VHKUHSOYCDOUVMN2HBL4JEXAGZS64K7M6L6DDIIF755TQ, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.248440564 (929h59m9.119523619s) [noise=true , address=145.79.227.31:4799 , piece_id=NOBWZTUK4VDKXM3PCO3Y623KIW2XKFRZXWZEDFVVRT5JHTPCZZQQ, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.249844076 (929h59m9.120927121s) [noise=true , address=145.79.227.50:4373 , piece_id=R5A7RCLHAH3ODMP563M27O6T7ZNACL36HJJOFVLRHYG456XUXXBA, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.251275041 (929h59m9.122358076s) [noise=true , address=145.79.227.52:4447 , piece_id=ZZMECYYJKEXTVZI6WQVEMT23OAVZ7SILIHSSE2OHFKSESUAQHJRQ, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.252655870 (929h59m9.123738915s) [noise=true , address=145.79.227.21:4473 , piece_id=FHOGGYMZY7DU6T7FUFNALWHI4Y6SDMPYF242QLW7C4UZIKKXEJ3A, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.253473537 (929h59m9.124556582s) [noise=true , address=145.79.227.13:4803 , piece_id=DNNAZPZ2TYQI6VCXNIZKSMU5VQCXKEDN3L6K6MHSVNRN3R246BCQ, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.254758193 (929h59m9.125841238s) [noise=true , address=145.79.227.51:4409 , piece_id=3FOIHL6K6JAEDPWIRHT7IJOQ534UUKYAVCNAW4V3R5QL3D25UKAA, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:11.087868620 (929h59m10.958951665s) [upload=813972]: Next returned nil segment Jan 13 09:28:11.087884400 (929h59m10.958967435s) [upload=813972]: Waiting for error group managing segments... Jan 13 09:28:11.388729734 (929h59m11.259812799s) [noise=true , address=145.79.227.50:4373 , piece_id=R5A7RCLHAH3ODMP563M27O6T7ZNACL36HJJOFVLRHYG456XUXXBA, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.399388468 (929h59m11.270471504s) [noise=true , address=145.79.227.21:4473 , piece_id=FHOGGYMZY7DU6T7FUFNALWHI4Y6SDMPYF242QLW7C4UZIKKXEJ3A, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.405180828 (929h59m11.276263873s) [noise=true , address=145.79.227.13:4803 , piece_id=DNNAZPZ2TYQI6VCXNIZKSMU5VQCXKEDN3L6K6MHSVNRN3R246BCQ, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.423915039 (929h59m11.294998074s) [noise=true , address=145.79.227.31:4799 , piece_id=NOBWZTUK4VDKXM3PCO3Y623KIW2XKFRZXWZEDFVVRT5JHTPCZZQQ, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.425205065 (929h59m11.29628811s) [noise=true , address=145.79.227.48:4301 , piece_id=BIBJBBTITB6AA2SWXXZAIEOOAIARHUIVCBXHB4JXMUSELMZSLLTA, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.427897582 (929h59m11.298980617s) [noise=true , address=145.79.227.5:4221 , piece_id=MAYWAFPWGGY6OJMP2OPGXJ4CFWMHIOL32IDBYGFVIONN6PFRDK2Q, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.461421214 (929h59m11.332504249s) [noise=true , address=145.79.227.2:4113 , piece_id=KUG2JKJTNJWMTZAP5F4HGA2UEKRHORRYVFKQ2FAX4KLQDVEX7UYQ, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.466681119 (929h59m11.337764154s) [noise=true , address=145.79.227.37:4566 , piece_id=NHE62LRK3ZLPFLQXOIPJZ3LB47RSNU7L2C3DGJKZYV3EXLYBSN6Q, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.468074292 (929h59m11.339157327s) [noise=true , address=145.79.227.11:4732 , piece_id=ZNFPDXJOGVKRQHPN66JHECSZLNEQ57MVWX4GAA6ETQSCNWCJQ75Q, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.472717633 (929h59m11.343800668s) [noise=true , address=145.79.227.35:4483 , piece_id=PH7UUSSVGW3I2KXRBDWEIFHDL3SC63SYDUPWMKG3NX3HKG6E3OYA, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.502968303 (929h59m11.374051338s) [noise=true , address=145.79.227.7:4293 , piece_id=LTQ6DD2KQ55UCWIJ43SWJO7Q5IDDJFTSD2Z7SVMHR2HLUOUSBNDQ, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.554022174 (929h59m11.425105219s) [noise=true , address=145.79.227.51:4409 , piece_id=3FOIHL6K6JAEDPWIRHT7IJOQ534UUKYAVCNAW4V3R5QL3D25UKAA, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.585092324 (929h59m11.456175369s) [noise=true , address=145.79.227.47:4267 , piece_id=RB3RDZAJRF3VV6LL6LUI5544BM7MFTGUIQ64Y4G23FHFRIYWZNUA, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.585114155 (929h59m11.4561972s) [seg_pos={0 0}, upload=813972]: Segment reached optimal threshold of 13 pieces. Jan 13 09:28:11.585293828 (929h59m11.456376863s) [noise=true , address=145.79.227.52:4447 , piece_id=ZZMECYYJKEXTVZI6WQVEMT23OAVZ7SILIHSSE2OHFKSESUAQHJRQ, seg_pos={0 0}, upload=813972]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP): protocol: write tcp 188.166.225.170:44548->145.79.227.52:4447: use of closed network connection; write tcp 188.166.225.170:44548->145.79.227.52:4447: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44548->145.79.227.52:4447: use of closed network connection Jan 13 09:28:11.585299989 (929h59m11.456383025s) [noise=true , address=145.79.227.52:4447 , piece_id=ZZMECYYJKEXTVZI6WQVEMT23OAVZ7SILIHSSE2OHFKSESUAQHJRQ, seg_pos={0 0}, upload=813972]: Long tail cancellation due to optimal threshold Jan 13 09:28:11.585433102 (929h59m11.456516137s) [noise=true , address=145.79.227.6:4257 , piece_id=7NPEYE7VHKUHSOYCDOUVMN2HBL4JEXAGZS64K7M6L6DDIIF755TQ, seg_pos={0 0}, upload=813972]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK): protocol: write tcp 188.166.225.170:52450->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:52450->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52450->145.79.227.6:4257: use of closed network connection Jan 13 09:28:11.585440497 (929h59m11.456523532s) [noise=true , address=145.79.227.6:4257 , piece_id=7NPEYE7VHKUHSOYCDOUVMN2HBL4JEXAGZS64K7M6L6DDIIF755TQ, seg_pos={0 0}, upload=813972]: Long tail cancellation due to optimal threshold Jan 13 09:28:11.585617424 (929h59m11.456700469s) [noise=true , address=145.79.227.49:4337 , piece_id=ZRIX2M4OOA6LKJ6FJ2JORHE4XZQDJRG3KSBUNRJ3EEEXMIK5VMYA, seg_pos={0 0}, upload=813972]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: read tcp 188.166.225.170:49630->145.79.227.49:4337: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 13 09:28:11.585629516 (929h59m11.456712562s) [noise=true , address=145.79.227.49:4337 , piece_id=ZRIX2M4OOA6LKJ6FJ2JORHE4XZQDJRG3KSBUNRJ3EEEXMIK5VMYA, seg_pos={0 0}, upload=813972]: Long tail cancellation due to optimal threshold Jan 13 09:28:11.585663561 (929h59m11.456746596s) [seg_pos={0 0}, upload=813972]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 13 09:28:11.585783911 (929h59m11.456866946s) [upload=813972]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 13 09:28:11.585791646 (929h59m11.456874681s) [upload=813972]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 13 09:28:11.889773771 (929h59m11.760856816s) [upload=813972]: Done uploading segments... Jan 13 09:28:11.889820049 (929h59m11.760903094s) [upload=813972]: Upload finished. err: <nil> Duration: 5.815964154s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 10.640122215s trace: https://jaeger.dev.storj.io/trace/4226536f67e7860 FAIL (2026-01-13 10:01:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 10.640151069s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 59 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001328214s trace: https://jaeger.dev.storj.io/trace/6c9d0c5c5580518b FAIL (2026-01-13 10:01:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001419307s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 5.942372208s trace: https://jaeger.dev.storj.io/trace/209f6f5543c7dce4 FAIL (2026-01-13 10:01:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: NOBWZTUK4VDKXM3PCO3Y623KIW2XKFRZXWZEDFVVRT5JHTPCZZQQ) context deadline exceeded; piecestore: (Node ID: 12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj, Piece ID: 3FOIHL6K6JAEDPWIRHT7IJOQ534UUKYAVCNAW4V3R5QL3D25UKAA) context deadline exceeded; piecestore: (Node ID: 1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, Piece ID: KUG2JKJTNJWMTZAP5F4HGA2UEKRHORRYVFKQ2FAX4KLQDVEX7UYQ) context deadline exceeded; piecestore: (Node ID: 12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, Piece ID: LTQ6DD2KQ55UCWIJ43SWJO7Q5IDDJFTSD2Z7SVMHR2HLUOUSBNDQ) context deadline exceeded ======LOGS====== ent. Inline: false Jan 13 09:28:06.315478350 (929h59m6.186561385s) [upload=813972]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 13 09:28:06.315483540 (929h59m6.186566575s) [upload=813972]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 13 09:28:09.241621500 (929h59m9.112704545s) [seg_pos={0 0}, upload=813972]: Begin upload segment... Jan 13 09:28:09.241671425 (929h59m9.11275446s) [seg_pos={0 0}, upload=813972]: Done begin upload segment. Jan 13 09:28:09.241703947 (929h59m9.112786992s) [noise=true , address=145.79.227.47:4267 , piece_id=RB3RDZAJRF3VV6LL6LUI5544BM7MFTGUIQ64Y4G23FHFRIYWZNUA, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.241712664 (929h59m9.112795699s) [noise=true , address=145.79.227.49:4337 , piece_id=ZRIX2M4OOA6LKJ6FJ2JORHE4XZQDJRG3KSBUNRJ3EEEXMIK5VMYA, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.242438585 (929h59m9.11352162s) [noise=true , address=145.79.227.2:4113 , piece_id=KUG2JKJTNJWMTZAP5F4HGA2UEKRHORRYVFKQ2FAX4KLQDVEX7UYQ, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.242727556 (929h59m9.113810591s) [noise=true , address=145.79.227.35:4483 , piece_id=PH7UUSSVGW3I2KXRBDWEIFHDL3SC63SYDUPWMKG3NX3HKG6E3OYA, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.243047555 (929h59m9.1141306s) [noise=true , address=145.79.227.5:4221 , piece_id=MAYWAFPWGGY6OJMP2OPGXJ4CFWMHIOL32IDBYGFVIONN6PFRDK2Q, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.243340904 (929h59m9.114423939s) [noise=true , address=145.79.227.48:4301 , piece_id=BIBJBBTITB6AA2SWXXZAIEOOAIARHUIVCBXHB4JXMUSELMZSLLTA, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.243735925 (929h59m9.11481897s) [noise=true , address=145.79.227.37:4566 , piece_id=NHE62LRK3ZLPFLQXOIPJZ3LB47RSNU7L2C3DGJKZYV3EXLYBSN6Q, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.244030276 (929h59m9.115113312s) [noise=true , address=145.79.227.11:4732 , piece_id=ZNFPDXJOGVKRQHPN66JHECSZLNEQ57MVWX4GAA6ETQSCNWCJQ75Q, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.245863306 (929h59m9.116946341s) [noise=true , address=145.79.227.7:4293 , piece_id=LTQ6DD2KQ55UCWIJ43SWJO7Q5IDDJFTSD2Z7SVMHR2HLUOUSBNDQ, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.247027153 (929h59m9.118110188s) [noise=true , address=145.79.227.6:4257 , piece_id=7NPEYE7VHKUHSOYCDOUVMN2HBL4JEXAGZS64K7M6L6DDIIF755TQ, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.248440564 (929h59m9.119523619s) [noise=true , address=145.79.227.31:4799 , piece_id=NOBWZTUK4VDKXM3PCO3Y623KIW2XKFRZXWZEDFVVRT5JHTPCZZQQ, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.249844076 (929h59m9.120927121s) [noise=true , address=145.79.227.50:4373 , piece_id=R5A7RCLHAH3ODMP563M27O6T7ZNACL36HJJOFVLRHYG456XUXXBA, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.251275041 (929h59m9.122358076s) [noise=true , address=145.79.227.52:4447 , piece_id=ZZMECYYJKEXTVZI6WQVEMT23OAVZ7SILIHSSE2OHFKSESUAQHJRQ, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.252655870 (929h59m9.123738915s) [noise=true , address=145.79.227.21:4473 , piece_id=FHOGGYMZY7DU6T7FUFNALWHI4Y6SDMPYF242QLW7C4UZIKKXEJ3A, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.253473537 (929h59m9.124556582s) [noise=true , address=145.79.227.13:4803 , piece_id=DNNAZPZ2TYQI6VCXNIZKSMU5VQCXKEDN3L6K6MHSVNRN3R246BCQ, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.254758193 (929h59m9.125841238s) [noise=true , address=145.79.227.51:4409 , piece_id=3FOIHL6K6JAEDPWIRHT7IJOQ534UUKYAVCNAW4V3R5QL3D25UKAA, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:11.087868620 (929h59m10.958951665s) [upload=813972]: Next returned nil segment Jan 13 09:28:11.087884400 (929h59m10.958967435s) [upload=813972]: Waiting for error group managing segments... Jan 13 09:28:11.388729734 (929h59m11.259812799s) [noise=true , address=145.79.227.50:4373 , piece_id=R5A7RCLHAH3ODMP563M27O6T7ZNACL36HJJOFVLRHYG456XUXXBA, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.399388468 (929h59m11.270471504s) [noise=true , address=145.79.227.21:4473 , piece_id=FHOGGYMZY7DU6T7FUFNALWHI4Y6SDMPYF242QLW7C4UZIKKXEJ3A, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.405180828 (929h59m11.276263873s) [noise=true , address=145.79.227.13:4803 , piece_id=DNNAZPZ2TYQI6VCXNIZKSMU5VQCXKEDN3L6K6MHSVNRN3R246BCQ, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.423915039 (929h59m11.294998074s) [noise=true , address=145.79.227.31:4799 , piece_id=NOBWZTUK4VDKXM3PCO3Y623KIW2XKFRZXWZEDFVVRT5JHTPCZZQQ, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.425205065 (929h59m11.29628811s) [noise=true , address=145.79.227.48:4301 , piece_id=BIBJBBTITB6AA2SWXXZAIEOOAIARHUIVCBXHB4JXMUSELMZSLLTA, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.427897582 (929h59m11.298980617s) [noise=true , address=145.79.227.5:4221 , piece_id=MAYWAFPWGGY6OJMP2OPGXJ4CFWMHIOL32IDBYGFVIONN6PFRDK2Q, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.461421214 (929h59m11.332504249s) [noise=true , address=145.79.227.2:4113 , piece_id=KUG2JKJTNJWMTZAP5F4HGA2UEKRHORRYVFKQ2FAX4KLQDVEX7UYQ, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.466681119 (929h59m11.337764154s) [noise=true , address=145.79.227.37:4566 , piece_id=NHE62LRK3ZLPFLQXOIPJZ3LB47RSNU7L2C3DGJKZYV3EXLYBSN6Q, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.468074292 (929h59m11.339157327s) [noise=true , address=145.79.227.11:4732 , piece_id=ZNFPDXJOGVKRQHPN66JHECSZLNEQ57MVWX4GAA6ETQSCNWCJQ75Q, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.472717633 (929h59m11.343800668s) [noise=true , address=145.79.227.35:4483 , piece_id=PH7UUSSVGW3I2KXRBDWEIFHDL3SC63SYDUPWMKG3NX3HKG6E3OYA, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.502968303 (929h59m11.374051338s) [noise=true , address=145.79.227.7:4293 , piece_id=LTQ6DD2KQ55UCWIJ43SWJO7Q5IDDJFTSD2Z7SVMHR2HLUOUSBNDQ, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.554022174 (929h59m11.425105219s) [noise=true , address=145.79.227.51:4409 , piece_id=3FOIHL6K6JAEDPWIRHT7IJOQ534UUKYAVCNAW4V3R5QL3D25UKAA, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.585092324 (929h59m11.456175369s) [noise=true , address=145.79.227.47:4267 , piece_id=RB3RDZAJRF3VV6LL6LUI5544BM7MFTGUIQ64Y4G23FHFRIYWZNUA, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.585114155 (929h59m11.4561972s) [seg_pos={0 0}, upload=813972]: Segment reached optimal threshold of 13 pieces. Jan 13 09:28:11.585293828 (929h59m11.456376863s) [noise=true , address=145.79.227.52:4447 , piece_id=ZZMECYYJKEXTVZI6WQVEMT23OAVZ7SILIHSSE2OHFKSESUAQHJRQ, seg_pos={0 0}, upload=813972]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP): protocol: write tcp 188.166.225.170:44548->145.79.227.52:4447: use of closed network connection; write tcp 188.166.225.170:44548->145.79.227.52:4447: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44548->145.79.227.52:4447: use of closed network connection Jan 13 09:28:11.585299989 (929h59m11.456383025s) [noise=true , address=145.79.227.52:4447 , piece_id=ZZMECYYJKEXTVZI6WQVEMT23OAVZ7SILIHSSE2OHFKSESUAQHJRQ, seg_pos={0 0}, upload=813972]: Long tail cancellation due to optimal threshold Jan 13 09:28:11.585433102 (929h59m11.456516137s) [noise=true , address=145.79.227.6:4257 , piece_id=7NPEYE7VHKUHSOYCDOUVMN2HBL4JEXAGZS64K7M6L6DDIIF755TQ, seg_pos={0 0}, upload=813972]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK): protocol: write tcp 188.166.225.170:52450->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:52450->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52450->145.79.227.6:4257: use of closed network connection Jan 13 09:28:11.585440497 (929h59m11.456523532s) [noise=true , address=145.79.227.6:4257 , piece_id=7NPEYE7VHKUHSOYCDOUVMN2HBL4JEXAGZS64K7M6L6DDIIF755TQ, seg_pos={0 0}, upload=813972]: Long tail cancellation due to optimal threshold Jan 13 09:28:11.585617424 (929h59m11.456700469s) [noise=true , address=145.79.227.49:4337 , piece_id=ZRIX2M4OOA6LKJ6FJ2JORHE4XZQDJRG3KSBUNRJ3EEEXMIK5VMYA, seg_pos={0 0}, upload=813972]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: read tcp 188.166.225.170:49630->145.79.227.49:4337: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 13 09:28:11.585629516 (929h59m11.456712562s) [noise=true , address=145.79.227.49:4337 , piece_id=ZRIX2M4OOA6LKJ6FJ2JORHE4XZQDJRG3KSBUNRJ3EEEXMIK5VMYA, seg_pos={0 0}, upload=813972]: Long tail cancellation due to optimal threshold Jan 13 09:28:11.585663561 (929h59m11.456746596s) [seg_pos={0 0}, upload=813972]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 13 09:28:11.585783911 (929h59m11.456866946s) [upload=813972]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 13 09:28:11.585791646 (929h59m11.456874681s) [upload=813972]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 13 09:28:11.889773771 (929h59m11.760856816s) [upload=813972]: Done uploading segments... Jan 13 09:28:11.889820049 (929h59m11.760903094s) [upload=813972]: Upload finished. err: <nil> Duration: 5.942397214s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...............X....................X.......X..X....XX......] duration: 18.862958808s trace: https://jaeger.dev.storj.io/trace/1422bf86419b0b28 FAIL (2026-01-13 10:01:19 UTC) orbiter warning: context deadline exceeded Duration: 18.86298642s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 59 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXX] duration: 5.000965092s trace: https://jaeger.dev.storj.io/trace/4ac9e98c831a2071 FAIL (2026-01-13 10:01:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000988737s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 7.69827276s trace: https://jaeger.dev.storj.io/trace/4226536f67e7860 FAIL (2026-01-13 10:01:00 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.698298601s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 5.749683643s trace: https://jaeger.dev.storj.io/trace/209f6f5543c7dce4 FAIL (2026-01-13 10:00:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: NOBWZTUK4VDKXM3PCO3Y623KIW2XKFRZXWZEDFVVRT5JHTPCZZQQ) context deadline exceeded; piecestore: (Node ID: 12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj, Piece ID: 3FOIHL6K6JAEDPWIRHT7IJOQ534UUKYAVCNAW4V3R5QL3D25UKAA) context deadline exceeded; piecestore: (Node ID: 1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, Piece ID: KUG2JKJTNJWMTZAP5F4HGA2UEKRHORRYVFKQ2FAX4KLQDVEX7UYQ) context deadline exceeded; piecestore: (Node ID: 12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, Piece ID: LTQ6DD2KQ55UCWIJ43SWJO7Q5IDDJFTSD2Z7SVMHR2HLUOUSBNDQ) context deadline exceeded ======LOGS====== ent. Inline: false Jan 13 09:28:06.315478350 (929h59m6.186561385s) [upload=813972]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 13 09:28:06.315483540 (929h59m6.186566575s) [upload=813972]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 13 09:28:09.241621500 (929h59m9.112704545s) [seg_pos={0 0}, upload=813972]: Begin upload segment... Jan 13 09:28:09.241671425 (929h59m9.11275446s) [seg_pos={0 0}, upload=813972]: Done begin upload segment. Jan 13 09:28:09.241703947 (929h59m9.112786992s) [noise=true , address=145.79.227.47:4267 , piece_id=RB3RDZAJRF3VV6LL6LUI5544BM7MFTGUIQ64Y4G23FHFRIYWZNUA, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.241712664 (929h59m9.112795699s) [noise=true , address=145.79.227.49:4337 , piece_id=ZRIX2M4OOA6LKJ6FJ2JORHE4XZQDJRG3KSBUNRJ3EEEXMIK5VMYA, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.242438585 (929h59m9.11352162s) [noise=true , address=145.79.227.2:4113 , piece_id=KUG2JKJTNJWMTZAP5F4HGA2UEKRHORRYVFKQ2FAX4KLQDVEX7UYQ, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.242727556 (929h59m9.113810591s) [noise=true , address=145.79.227.35:4483 , piece_id=PH7UUSSVGW3I2KXRBDWEIFHDL3SC63SYDUPWMKG3NX3HKG6E3OYA, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.243047555 (929h59m9.1141306s) [noise=true , address=145.79.227.5:4221 , piece_id=MAYWAFPWGGY6OJMP2OPGXJ4CFWMHIOL32IDBYGFVIONN6PFRDK2Q, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.243340904 (929h59m9.114423939s) [noise=true , address=145.79.227.48:4301 , piece_id=BIBJBBTITB6AA2SWXXZAIEOOAIARHUIVCBXHB4JXMUSELMZSLLTA, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.243735925 (929h59m9.11481897s) [noise=true , address=145.79.227.37:4566 , piece_id=NHE62LRK3ZLPFLQXOIPJZ3LB47RSNU7L2C3DGJKZYV3EXLYBSN6Q, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.244030276 (929h59m9.115113312s) [noise=true , address=145.79.227.11:4732 , piece_id=ZNFPDXJOGVKRQHPN66JHECSZLNEQ57MVWX4GAA6ETQSCNWCJQ75Q, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.245863306 (929h59m9.116946341s) [noise=true , address=145.79.227.7:4293 , piece_id=LTQ6DD2KQ55UCWIJ43SWJO7Q5IDDJFTSD2Z7SVMHR2HLUOUSBNDQ, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.247027153 (929h59m9.118110188s) [noise=true , address=145.79.227.6:4257 , piece_id=7NPEYE7VHKUHSOYCDOUVMN2HBL4JEXAGZS64K7M6L6DDIIF755TQ, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.248440564 (929h59m9.119523619s) [noise=true , address=145.79.227.31:4799 , piece_id=NOBWZTUK4VDKXM3PCO3Y623KIW2XKFRZXWZEDFVVRT5JHTPCZZQQ, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.249844076 (929h59m9.120927121s) [noise=true , address=145.79.227.50:4373 , piece_id=R5A7RCLHAH3ODMP563M27O6T7ZNACL36HJJOFVLRHYG456XUXXBA, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.251275041 (929h59m9.122358076s) [noise=true , address=145.79.227.52:4447 , piece_id=ZZMECYYJKEXTVZI6WQVEMT23OAVZ7SILIHSSE2OHFKSESUAQHJRQ, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.252655870 (929h59m9.123738915s) [noise=true , address=145.79.227.21:4473 , piece_id=FHOGGYMZY7DU6T7FUFNALWHI4Y6SDMPYF242QLW7C4UZIKKXEJ3A, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.253473537 (929h59m9.124556582s) [noise=true , address=145.79.227.13:4803 , piece_id=DNNAZPZ2TYQI6VCXNIZKSMU5VQCXKEDN3L6K6MHSVNRN3R246BCQ, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.254758193 (929h59m9.125841238s) [noise=true , address=145.79.227.51:4409 , piece_id=3FOIHL6K6JAEDPWIRHT7IJOQ534UUKYAVCNAW4V3R5QL3D25UKAA, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:11.087868620 (929h59m10.958951665s) [upload=813972]: Next returned nil segment Jan 13 09:28:11.087884400 (929h59m10.958967435s) [upload=813972]: Waiting for error group managing segments... Jan 13 09:28:11.388729734 (929h59m11.259812799s) [noise=true , address=145.79.227.50:4373 , piece_id=R5A7RCLHAH3ODMP563M27O6T7ZNACL36HJJOFVLRHYG456XUXXBA, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.399388468 (929h59m11.270471504s) [noise=true , address=145.79.227.21:4473 , piece_id=FHOGGYMZY7DU6T7FUFNALWHI4Y6SDMPYF242QLW7C4UZIKKXEJ3A, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.405180828 (929h59m11.276263873s) [noise=true , address=145.79.227.13:4803 , piece_id=DNNAZPZ2TYQI6VCXNIZKSMU5VQCXKEDN3L6K6MHSVNRN3R246BCQ, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.423915039 (929h59m11.294998074s) [noise=true , address=145.79.227.31:4799 , piece_id=NOBWZTUK4VDKXM3PCO3Y623KIW2XKFRZXWZEDFVVRT5JHTPCZZQQ, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.425205065 (929h59m11.29628811s) [noise=true , address=145.79.227.48:4301 , piece_id=BIBJBBTITB6AA2SWXXZAIEOOAIARHUIVCBXHB4JXMUSELMZSLLTA, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.427897582 (929h59m11.298980617s) [noise=true , address=145.79.227.5:4221 , piece_id=MAYWAFPWGGY6OJMP2OPGXJ4CFWMHIOL32IDBYGFVIONN6PFRDK2Q, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.461421214 (929h59m11.332504249s) [noise=true , address=145.79.227.2:4113 , piece_id=KUG2JKJTNJWMTZAP5F4HGA2UEKRHORRYVFKQ2FAX4KLQDVEX7UYQ, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.466681119 (929h59m11.337764154s) [noise=true , address=145.79.227.37:4566 , piece_id=NHE62LRK3ZLPFLQXOIPJZ3LB47RSNU7L2C3DGJKZYV3EXLYBSN6Q, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.468074292 (929h59m11.339157327s) [noise=true , address=145.79.227.11:4732 , piece_id=ZNFPDXJOGVKRQHPN66JHECSZLNEQ57MVWX4GAA6ETQSCNWCJQ75Q, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.472717633 (929h59m11.343800668s) [noise=true , address=145.79.227.35:4483 , piece_id=PH7UUSSVGW3I2KXRBDWEIFHDL3SC63SYDUPWMKG3NX3HKG6E3OYA, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.502968303 (929h59m11.374051338s) [noise=true , address=145.79.227.7:4293 , piece_id=LTQ6DD2KQ55UCWIJ43SWJO7Q5IDDJFTSD2Z7SVMHR2HLUOUSBNDQ, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.554022174 (929h59m11.425105219s) [noise=true , address=145.79.227.51:4409 , piece_id=3FOIHL6K6JAEDPWIRHT7IJOQ534UUKYAVCNAW4V3R5QL3D25UKAA, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.585092324 (929h59m11.456175369s) [noise=true , address=145.79.227.47:4267 , piece_id=RB3RDZAJRF3VV6LL6LUI5544BM7MFTGUIQ64Y4G23FHFRIYWZNUA, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.585114155 (929h59m11.4561972s) [seg_pos={0 0}, upload=813972]: Segment reached optimal threshold of 13 pieces. Jan 13 09:28:11.585293828 (929h59m11.456376863s) [noise=true , address=145.79.227.52:4447 , piece_id=ZZMECYYJKEXTVZI6WQVEMT23OAVZ7SILIHSSE2OHFKSESUAQHJRQ, seg_pos={0 0}, upload=813972]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP): protocol: write tcp 188.166.225.170:44548->145.79.227.52:4447: use of closed network connection; write tcp 188.166.225.170:44548->145.79.227.52:4447: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44548->145.79.227.52:4447: use of closed network connection Jan 13 09:28:11.585299989 (929h59m11.456383025s) [noise=true , address=145.79.227.52:4447 , piece_id=ZZMECYYJKEXTVZI6WQVEMT23OAVZ7SILIHSSE2OHFKSESUAQHJRQ, seg_pos={0 0}, upload=813972]: Long tail cancellation due to optimal threshold Jan 13 09:28:11.585433102 (929h59m11.456516137s) [noise=true , address=145.79.227.6:4257 , piece_id=7NPEYE7VHKUHSOYCDOUVMN2HBL4JEXAGZS64K7M6L6DDIIF755TQ, seg_pos={0 0}, upload=813972]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK): protocol: write tcp 188.166.225.170:52450->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:52450->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52450->145.79.227.6:4257: use of closed network connection Jan 13 09:28:11.585440497 (929h59m11.456523532s) [noise=true , address=145.79.227.6:4257 , piece_id=7NPEYE7VHKUHSOYCDOUVMN2HBL4JEXAGZS64K7M6L6DDIIF755TQ, seg_pos={0 0}, upload=813972]: Long tail cancellation due to optimal threshold Jan 13 09:28:11.585617424 (929h59m11.456700469s) [noise=true , address=145.79.227.49:4337 , piece_id=ZRIX2M4OOA6LKJ6FJ2JORHE4XZQDJRG3KSBUNRJ3EEEXMIK5VMYA, seg_pos={0 0}, upload=813972]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: read tcp 188.166.225.170:49630->145.79.227.49:4337: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 13 09:28:11.585629516 (929h59m11.456712562s) [noise=true , address=145.79.227.49:4337 , piece_id=ZRIX2M4OOA6LKJ6FJ2JORHE4XZQDJRG3KSBUNRJ3EEEXMIK5VMYA, seg_pos={0 0}, upload=813972]: Long tail cancellation due to optimal threshold Jan 13 09:28:11.585663561 (929h59m11.456746596s) [seg_pos={0 0}, upload=813972]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 13 09:28:11.585783911 (929h59m11.456866946s) [upload=813972]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 13 09:28:11.585791646 (929h59m11.456874681s) [upload=813972]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 13 09:28:11.889773771 (929h59m11.760856816s) [upload=813972]: Done uploading segments... Jan 13 09:28:11.889820049 (929h59m11.760903094s) [upload=813972]: Upload finished. err: <nil> Duration: 5.749713209s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.574283047s trace: https://jaeger.dev.storj.io/trace/4226536f67e7860 FAIL (2026-01-13 10:00:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.574337109s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [................X....................X.......X..X....XX.....] duration: 26.102715809s trace: https://jaeger.dev.storj.io/trace/1422bf86419b0b28 FAIL (2026-01-13 10:00:21 UTC) orbiter warning: context deadline exceeded Duration: 26.102766736s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 30.035078787s trace: https://jaeger.dev.storj.io/trace/4226536f67e7860 FAIL (2026-01-13 10:00:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.035103344s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 59 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXX] duration: 5.001817198s trace: https://jaeger.dev.storj.io/trace/1121835c1d0b7c21 FAIL (2026-01-13 10:00:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001842755s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 59 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXX] duration: 5.001445157s trace: https://jaeger.dev.storj.io/trace/68ac16fdc1112677 FAIL (2026-01-13 09:59:48 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001466489s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.684552739s trace: https://jaeger.dev.storj.io/trace/209f6f5543c7dce4 FAIL (2026-01-13 09:59:38 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: NOBWZTUK4VDKXM3PCO3Y623KIW2XKFRZXWZEDFVVRT5JHTPCZZQQ) context deadline exceeded; piecestore: (Node ID: 12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj, Piece ID: 3FOIHL6K6JAEDPWIRHT7IJOQ534UUKYAVCNAW4V3R5QL3D25UKAA) context deadline exceeded; piecestore: (Node ID: 1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, Piece ID: KUG2JKJTNJWMTZAP5F4HGA2UEKRHORRYVFKQ2FAX4KLQDVEX7UYQ) context deadline exceeded; piecestore: (Node ID: 12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, Piece ID: LTQ6DD2KQ55UCWIJ43SWJO7Q5IDDJFTSD2Z7SVMHR2HLUOUSBNDQ) context deadline exceeded ======LOGS====== ent. Inline: false Jan 13 09:28:06.315478350 (929h59m6.186561385s) [upload=813972]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 13 09:28:06.315483540 (929h59m6.186566575s) [upload=813972]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 13 09:28:09.241621500 (929h59m9.112704545s) [seg_pos={0 0}, upload=813972]: Begin upload segment... Jan 13 09:28:09.241671425 (929h59m9.11275446s) [seg_pos={0 0}, upload=813972]: Done begin upload segment. Jan 13 09:28:09.241703947 (929h59m9.112786992s) [noise=true , address=145.79.227.47:4267 , piece_id=RB3RDZAJRF3VV6LL6LUI5544BM7MFTGUIQ64Y4G23FHFRIYWZNUA, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.241712664 (929h59m9.112795699s) [noise=true , address=145.79.227.49:4337 , piece_id=ZRIX2M4OOA6LKJ6FJ2JORHE4XZQDJRG3KSBUNRJ3EEEXMIK5VMYA, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.242438585 (929h59m9.11352162s) [noise=true , address=145.79.227.2:4113 , piece_id=KUG2JKJTNJWMTZAP5F4HGA2UEKRHORRYVFKQ2FAX4KLQDVEX7UYQ, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.242727556 (929h59m9.113810591s) [noise=true , address=145.79.227.35:4483 , piece_id=PH7UUSSVGW3I2KXRBDWEIFHDL3SC63SYDUPWMKG3NX3HKG6E3OYA, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.243047555 (929h59m9.1141306s) [noise=true , address=145.79.227.5:4221 , piece_id=MAYWAFPWGGY6OJMP2OPGXJ4CFWMHIOL32IDBYGFVIONN6PFRDK2Q, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.243340904 (929h59m9.114423939s) [noise=true , address=145.79.227.48:4301 , piece_id=BIBJBBTITB6AA2SWXXZAIEOOAIARHUIVCBXHB4JXMUSELMZSLLTA, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.243735925 (929h59m9.11481897s) [noise=true , address=145.79.227.37:4566 , piece_id=NHE62LRK3ZLPFLQXOIPJZ3LB47RSNU7L2C3DGJKZYV3EXLYBSN6Q, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.244030276 (929h59m9.115113312s) [noise=true , address=145.79.227.11:4732 , piece_id=ZNFPDXJOGVKRQHPN66JHECSZLNEQ57MVWX4GAA6ETQSCNWCJQ75Q, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.245863306 (929h59m9.116946341s) [noise=true , address=145.79.227.7:4293 , piece_id=LTQ6DD2KQ55UCWIJ43SWJO7Q5IDDJFTSD2Z7SVMHR2HLUOUSBNDQ, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.247027153 (929h59m9.118110188s) [noise=true , address=145.79.227.6:4257 , piece_id=7NPEYE7VHKUHSOYCDOUVMN2HBL4JEXAGZS64K7M6L6DDIIF755TQ, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.248440564 (929h59m9.119523619s) [noise=true , address=145.79.227.31:4799 , piece_id=NOBWZTUK4VDKXM3PCO3Y623KIW2XKFRZXWZEDFVVRT5JHTPCZZQQ, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.249844076 (929h59m9.120927121s) [noise=true , address=145.79.227.50:4373 , piece_id=R5A7RCLHAH3ODMP563M27O6T7ZNACL36HJJOFVLRHYG456XUXXBA, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.251275041 (929h59m9.122358076s) [noise=true , address=145.79.227.52:4447 , piece_id=ZZMECYYJKEXTVZI6WQVEMT23OAVZ7SILIHSSE2OHFKSESUAQHJRQ, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.252655870 (929h59m9.123738915s) [noise=true , address=145.79.227.21:4473 , piece_id=FHOGGYMZY7DU6T7FUFNALWHI4Y6SDMPYF242QLW7C4UZIKKXEJ3A, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.253473537 (929h59m9.124556582s) [noise=true , address=145.79.227.13:4803 , piece_id=DNNAZPZ2TYQI6VCXNIZKSMU5VQCXKEDN3L6K6MHSVNRN3R246BCQ, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.254758193 (929h59m9.125841238s) [noise=true , address=145.79.227.51:4409 , piece_id=3FOIHL6K6JAEDPWIRHT7IJOQ534UUKYAVCNAW4V3R5QL3D25UKAA, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:11.087868620 (929h59m10.958951665s) [upload=813972]: Next returned nil segment Jan 13 09:28:11.087884400 (929h59m10.958967435s) [upload=813972]: Waiting for error group managing segments... Jan 13 09:28:11.388729734 (929h59m11.259812799s) [noise=true , address=145.79.227.50:4373 , piece_id=R5A7RCLHAH3ODMP563M27O6T7ZNACL36HJJOFVLRHYG456XUXXBA, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.399388468 (929h59m11.270471504s) [noise=true , address=145.79.227.21:4473 , piece_id=FHOGGYMZY7DU6T7FUFNALWHI4Y6SDMPYF242QLW7C4UZIKKXEJ3A, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.405180828 (929h59m11.276263873s) [noise=true , address=145.79.227.13:4803 , piece_id=DNNAZPZ2TYQI6VCXNIZKSMU5VQCXKEDN3L6K6MHSVNRN3R246BCQ, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.423915039 (929h59m11.294998074s) [noise=true , address=145.79.227.31:4799 , piece_id=NOBWZTUK4VDKXM3PCO3Y623KIW2XKFRZXWZEDFVVRT5JHTPCZZQQ, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.425205065 (929h59m11.29628811s) [noise=true , address=145.79.227.48:4301 , piece_id=BIBJBBTITB6AA2SWXXZAIEOOAIARHUIVCBXHB4JXMUSELMZSLLTA, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.427897582 (929h59m11.298980617s) [noise=true , address=145.79.227.5:4221 , piece_id=MAYWAFPWGGY6OJMP2OPGXJ4CFWMHIOL32IDBYGFVIONN6PFRDK2Q, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.461421214 (929h59m11.332504249s) [noise=true , address=145.79.227.2:4113 , piece_id=KUG2JKJTNJWMTZAP5F4HGA2UEKRHORRYVFKQ2FAX4KLQDVEX7UYQ, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.466681119 (929h59m11.337764154s) [noise=true , address=145.79.227.37:4566 , piece_id=NHE62LRK3ZLPFLQXOIPJZ3LB47RSNU7L2C3DGJKZYV3EXLYBSN6Q, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.468074292 (929h59m11.339157327s) [noise=true , address=145.79.227.11:4732 , piece_id=ZNFPDXJOGVKRQHPN66JHECSZLNEQ57MVWX4GAA6ETQSCNWCJQ75Q, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.472717633 (929h59m11.343800668s) [noise=true , address=145.79.227.35:4483 , piece_id=PH7UUSSVGW3I2KXRBDWEIFHDL3SC63SYDUPWMKG3NX3HKG6E3OYA, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.502968303 (929h59m11.374051338s) [noise=true , address=145.79.227.7:4293 , piece_id=LTQ6DD2KQ55UCWIJ43SWJO7Q5IDDJFTSD2Z7SVMHR2HLUOUSBNDQ, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.554022174 (929h59m11.425105219s) [noise=true , address=145.79.227.51:4409 , piece_id=3FOIHL6K6JAEDPWIRHT7IJOQ534UUKYAVCNAW4V3R5QL3D25UKAA, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.585092324 (929h59m11.456175369s) [noise=true , address=145.79.227.47:4267 , piece_id=RB3RDZAJRF3VV6LL6LUI5544BM7MFTGUIQ64Y4G23FHFRIYWZNUA, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.585114155 (929h59m11.4561972s) [seg_pos={0 0}, upload=813972]: Segment reached optimal threshold of 13 pieces. Jan 13 09:28:11.585293828 (929h59m11.456376863s) [noise=true , address=145.79.227.52:4447 , piece_id=ZZMECYYJKEXTVZI6WQVEMT23OAVZ7SILIHSSE2OHFKSESUAQHJRQ, seg_pos={0 0}, upload=813972]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP): protocol: write tcp 188.166.225.170:44548->145.79.227.52:4447: use of closed network connection; write tcp 188.166.225.170:44548->145.79.227.52:4447: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44548->145.79.227.52:4447: use of closed network connection Jan 13 09:28:11.585299989 (929h59m11.456383025s) [noise=true , address=145.79.227.52:4447 , piece_id=ZZMECYYJKEXTVZI6WQVEMT23OAVZ7SILIHSSE2OHFKSESUAQHJRQ, seg_pos={0 0}, upload=813972]: Long tail cancellation due to optimal threshold Jan 13 09:28:11.585433102 (929h59m11.456516137s) [noise=true , address=145.79.227.6:4257 , piece_id=7NPEYE7VHKUHSOYCDOUVMN2HBL4JEXAGZS64K7M6L6DDIIF755TQ, seg_pos={0 0}, upload=813972]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK): protocol: write tcp 188.166.225.170:52450->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:52450->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52450->145.79.227.6:4257: use of closed network connection Jan 13 09:28:11.585440497 (929h59m11.456523532s) [noise=true , address=145.79.227.6:4257 , piece_id=7NPEYE7VHKUHSOYCDOUVMN2HBL4JEXAGZS64K7M6L6DDIIF755TQ, seg_pos={0 0}, upload=813972]: Long tail cancellation due to optimal threshold Jan 13 09:28:11.585617424 (929h59m11.456700469s) [noise=true , address=145.79.227.49:4337 , piece_id=ZRIX2M4OOA6LKJ6FJ2JORHE4XZQDJRG3KSBUNRJ3EEEXMIK5VMYA, seg_pos={0 0}, upload=813972]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: read tcp 188.166.225.170:49630->145.79.227.49:4337: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 13 09:28:11.585629516 (929h59m11.456712562s) [noise=true , address=145.79.227.49:4337 , piece_id=ZRIX2M4OOA6LKJ6FJ2JORHE4XZQDJRG3KSBUNRJ3EEEXMIK5VMYA, seg_pos={0 0}, upload=813972]: Long tail cancellation due to optimal threshold Jan 13 09:28:11.585663561 (929h59m11.456746596s) [seg_pos={0 0}, upload=813972]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 13 09:28:11.585783911 (929h59m11.456866946s) [upload=813972]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 13 09:28:11.585791646 (929h59m11.456874681s) [upload=813972]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 13 09:28:11.889773771 (929h59m11.760856816s) [upload=813972]: Done uploading segments... Jan 13 09:28:11.889820049 (929h59m11.760903094s) [upload=813972]: Upload finished. err: <nil> Duration: 5.684585019s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.................X....................X.......X..X....XX....] duration: 18.624825262s trace: https://jaeger.dev.storj.io/trace/1422bf86419b0b28 FAIL (2026-01-13 09:59:21 UTC) orbiter warning: context deadline exceeded Duration: 18.62485118s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 5.724567149s trace: https://jaeger.dev.storj.io/trace/209f6f5543c7dce4 FAIL (2026-01-13 09:59:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: NOBWZTUK4VDKXM3PCO3Y623KIW2XKFRZXWZEDFVVRT5JHTPCZZQQ) context deadline exceeded; piecestore: (Node ID: 12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj, Piece ID: 3FOIHL6K6JAEDPWIRHT7IJOQ534UUKYAVCNAW4V3R5QL3D25UKAA) context deadline exceeded; piecestore: (Node ID: 1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, Piece ID: KUG2JKJTNJWMTZAP5F4HGA2UEKRHORRYVFKQ2FAX4KLQDVEX7UYQ) context deadline exceeded; piecestore: (Node ID: 12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, Piece ID: LTQ6DD2KQ55UCWIJ43SWJO7Q5IDDJFTSD2Z7SVMHR2HLUOUSBNDQ) context deadline exceeded ======LOGS====== ent. Inline: false Jan 13 09:28:06.315478350 (929h59m6.186561385s) [upload=813972]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 13 09:28:06.315483540 (929h59m6.186566575s) [upload=813972]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 13 09:28:09.241621500 (929h59m9.112704545s) [seg_pos={0 0}, upload=813972]: Begin upload segment... Jan 13 09:28:09.241671425 (929h59m9.11275446s) [seg_pos={0 0}, upload=813972]: Done begin upload segment. Jan 13 09:28:09.241703947 (929h59m9.112786992s) [noise=true , address=145.79.227.47:4267 , piece_id=RB3RDZAJRF3VV6LL6LUI5544BM7MFTGUIQ64Y4G23FHFRIYWZNUA, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.241712664 (929h59m9.112795699s) [noise=true , address=145.79.227.49:4337 , piece_id=ZRIX2M4OOA6LKJ6FJ2JORHE4XZQDJRG3KSBUNRJ3EEEXMIK5VMYA, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.242438585 (929h59m9.11352162s) [noise=true , address=145.79.227.2:4113 , piece_id=KUG2JKJTNJWMTZAP5F4HGA2UEKRHORRYVFKQ2FAX4KLQDVEX7UYQ, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.242727556 (929h59m9.113810591s) [noise=true , address=145.79.227.35:4483 , piece_id=PH7UUSSVGW3I2KXRBDWEIFHDL3SC63SYDUPWMKG3NX3HKG6E3OYA, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.243047555 (929h59m9.1141306s) [noise=true , address=145.79.227.5:4221 , piece_id=MAYWAFPWGGY6OJMP2OPGXJ4CFWMHIOL32IDBYGFVIONN6PFRDK2Q, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.243340904 (929h59m9.114423939s) [noise=true , address=145.79.227.48:4301 , piece_id=BIBJBBTITB6AA2SWXXZAIEOOAIARHUIVCBXHB4JXMUSELMZSLLTA, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.243735925 (929h59m9.11481897s) [noise=true , address=145.79.227.37:4566 , piece_id=NHE62LRK3ZLPFLQXOIPJZ3LB47RSNU7L2C3DGJKZYV3EXLYBSN6Q, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.244030276 (929h59m9.115113312s) [noise=true , address=145.79.227.11:4732 , piece_id=ZNFPDXJOGVKRQHPN66JHECSZLNEQ57MVWX4GAA6ETQSCNWCJQ75Q, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.245863306 (929h59m9.116946341s) [noise=true , address=145.79.227.7:4293 , piece_id=LTQ6DD2KQ55UCWIJ43SWJO7Q5IDDJFTSD2Z7SVMHR2HLUOUSBNDQ, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.247027153 (929h59m9.118110188s) [noise=true , address=145.79.227.6:4257 , piece_id=7NPEYE7VHKUHSOYCDOUVMN2HBL4JEXAGZS64K7M6L6DDIIF755TQ, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.248440564 (929h59m9.119523619s) [noise=true , address=145.79.227.31:4799 , piece_id=NOBWZTUK4VDKXM3PCO3Y623KIW2XKFRZXWZEDFVVRT5JHTPCZZQQ, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.249844076 (929h59m9.120927121s) [noise=true , address=145.79.227.50:4373 , piece_id=R5A7RCLHAH3ODMP563M27O6T7ZNACL36HJJOFVLRHYG456XUXXBA, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.251275041 (929h59m9.122358076s) [noise=true , address=145.79.227.52:4447 , piece_id=ZZMECYYJKEXTVZI6WQVEMT23OAVZ7SILIHSSE2OHFKSESUAQHJRQ, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.252655870 (929h59m9.123738915s) [noise=true , address=145.79.227.21:4473 , piece_id=FHOGGYMZY7DU6T7FUFNALWHI4Y6SDMPYF242QLW7C4UZIKKXEJ3A, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.253473537 (929h59m9.124556582s) [noise=true , address=145.79.227.13:4803 , piece_id=DNNAZPZ2TYQI6VCXNIZKSMU5VQCXKEDN3L6K6MHSVNRN3R246BCQ, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.254758193 (929h59m9.125841238s) [noise=true , address=145.79.227.51:4409 , piece_id=3FOIHL6K6JAEDPWIRHT7IJOQ534UUKYAVCNAW4V3R5QL3D25UKAA, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:11.087868620 (929h59m10.958951665s) [upload=813972]: Next returned nil segment Jan 13 09:28:11.087884400 (929h59m10.958967435s) [upload=813972]: Waiting for error group managing segments... Jan 13 09:28:11.388729734 (929h59m11.259812799s) [noise=true , address=145.79.227.50:4373 , piece_id=R5A7RCLHAH3ODMP563M27O6T7ZNACL36HJJOFVLRHYG456XUXXBA, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.399388468 (929h59m11.270471504s) [noise=true , address=145.79.227.21:4473 , piece_id=FHOGGYMZY7DU6T7FUFNALWHI4Y6SDMPYF242QLW7C4UZIKKXEJ3A, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.405180828 (929h59m11.276263873s) [noise=true , address=145.79.227.13:4803 , piece_id=DNNAZPZ2TYQI6VCXNIZKSMU5VQCXKEDN3L6K6MHSVNRN3R246BCQ, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.423915039 (929h59m11.294998074s) [noise=true , address=145.79.227.31:4799 , piece_id=NOBWZTUK4VDKXM3PCO3Y623KIW2XKFRZXWZEDFVVRT5JHTPCZZQQ, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.425205065 (929h59m11.29628811s) [noise=true , address=145.79.227.48:4301 , piece_id=BIBJBBTITB6AA2SWXXZAIEOOAIARHUIVCBXHB4JXMUSELMZSLLTA, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.427897582 (929h59m11.298980617s) [noise=true , address=145.79.227.5:4221 , piece_id=MAYWAFPWGGY6OJMP2OPGXJ4CFWMHIOL32IDBYGFVIONN6PFRDK2Q, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.461421214 (929h59m11.332504249s) [noise=true , address=145.79.227.2:4113 , piece_id=KUG2JKJTNJWMTZAP5F4HGA2UEKRHORRYVFKQ2FAX4KLQDVEX7UYQ, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.466681119 (929h59m11.337764154s) [noise=true , address=145.79.227.37:4566 , piece_id=NHE62LRK3ZLPFLQXOIPJZ3LB47RSNU7L2C3DGJKZYV3EXLYBSN6Q, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.468074292 (929h59m11.339157327s) [noise=true , address=145.79.227.11:4732 , piece_id=ZNFPDXJOGVKRQHPN66JHECSZLNEQ57MVWX4GAA6ETQSCNWCJQ75Q, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.472717633 (929h59m11.343800668s) [noise=true , address=145.79.227.35:4483 , piece_id=PH7UUSSVGW3I2KXRBDWEIFHDL3SC63SYDUPWMKG3NX3HKG6E3OYA, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.502968303 (929h59m11.374051338s) [noise=true , address=145.79.227.7:4293 , piece_id=LTQ6DD2KQ55UCWIJ43SWJO7Q5IDDJFTSD2Z7SVMHR2HLUOUSBNDQ, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.554022174 (929h59m11.425105219s) [noise=true , address=145.79.227.51:4409 , piece_id=3FOIHL6K6JAEDPWIRHT7IJOQ534UUKYAVCNAW4V3R5QL3D25UKAA, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.585092324 (929h59m11.456175369s) [noise=true , address=145.79.227.47:4267 , piece_id=RB3RDZAJRF3VV6LL6LUI5544BM7MFTGUIQ64Y4G23FHFRIYWZNUA, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.585114155 (929h59m11.4561972s) [seg_pos={0 0}, upload=813972]: Segment reached optimal threshold of 13 pieces. Jan 13 09:28:11.585293828 (929h59m11.456376863s) [noise=true , address=145.79.227.52:4447 , piece_id=ZZMECYYJKEXTVZI6WQVEMT23OAVZ7SILIHSSE2OHFKSESUAQHJRQ, seg_pos={0 0}, upload=813972]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP): protocol: write tcp 188.166.225.170:44548->145.79.227.52:4447: use of closed network connection; write tcp 188.166.225.170:44548->145.79.227.52:4447: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44548->145.79.227.52:4447: use of closed network connection Jan 13 09:28:11.585299989 (929h59m11.456383025s) [noise=true , address=145.79.227.52:4447 , piece_id=ZZMECYYJKEXTVZI6WQVEMT23OAVZ7SILIHSSE2OHFKSESUAQHJRQ, seg_pos={0 0}, upload=813972]: Long tail cancellation due to optimal threshold Jan 13 09:28:11.585433102 (929h59m11.456516137s) [noise=true , address=145.79.227.6:4257 , piece_id=7NPEYE7VHKUHSOYCDOUVMN2HBL4JEXAGZS64K7M6L6DDIIF755TQ, seg_pos={0 0}, upload=813972]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK): protocol: write tcp 188.166.225.170:52450->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:52450->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52450->145.79.227.6:4257: use of closed network connection Jan 13 09:28:11.585440497 (929h59m11.456523532s) [noise=true , address=145.79.227.6:4257 , piece_id=7NPEYE7VHKUHSOYCDOUVMN2HBL4JEXAGZS64K7M6L6DDIIF755TQ, seg_pos={0 0}, upload=813972]: Long tail cancellation due to optimal threshold Jan 13 09:28:11.585617424 (929h59m11.456700469s) [noise=true , address=145.79.227.49:4337 , piece_id=ZRIX2M4OOA6LKJ6FJ2JORHE4XZQDJRG3KSBUNRJ3EEEXMIK5VMYA, seg_pos={0 0}, upload=813972]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: read tcp 188.166.225.170:49630->145.79.227.49:4337: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 13 09:28:11.585629516 (929h59m11.456712562s) [noise=true , address=145.79.227.49:4337 , piece_id=ZRIX2M4OOA6LKJ6FJ2JORHE4XZQDJRG3KSBUNRJ3EEEXMIK5VMYA, seg_pos={0 0}, upload=813972]: Long tail cancellation due to optimal threshold Jan 13 09:28:11.585663561 (929h59m11.456746596s) [seg_pos={0 0}, upload=813972]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 13 09:28:11.585783911 (929h59m11.456866946s) [upload=813972]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 13 09:28:11.585791646 (929h59m11.456874681s) [upload=813972]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 13 09:28:11.889773771 (929h59m11.760856816s) [upload=813972]: Done uploading segments... Jan 13 09:28:11.889820049 (929h59m11.760903094s) [upload=813972]: Upload finished. err: <nil> Duration: 5.724595342s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 58 out of 60 is failed (threshold=6) [.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXX] duration: 5.00168843s trace: https://jaeger.dev.storj.io/trace/22cce1965876fc0f FAIL (2026-01-13 09:59:14 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001710873s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..................X....................X.......X..X....XX...] duration: 23.420106376s trace: https://jaeger.dev.storj.io/trace/1422bf86419b0b28 FAIL (2026-01-13 09:58:43 UTC) orbiter warning: context deadline exceeded Duration: 23.420120853s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 7.608371824s trace: https://jaeger.dev.storj.io/trace/4226536f67e7860 FAIL (2026-01-13 09:58:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.608395929s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...................X....................X.......X..X....XX..] duration: 28.116076038s trace: https://jaeger.dev.storj.io/trace/1422bf86419b0b28 FAIL (2026-01-13 09:58:19 UTC) orbiter warning: context deadline exceeded Duration: 28.11610299s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 5.665758823s trace: https://jaeger.dev.storj.io/trace/209f6f5543c7dce4 FAIL (2026-01-13 09:58:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: NOBWZTUK4VDKXM3PCO3Y623KIW2XKFRZXWZEDFVVRT5JHTPCZZQQ) context deadline exceeded; piecestore: (Node ID: 12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj, Piece ID: 3FOIHL6K6JAEDPWIRHT7IJOQ534UUKYAVCNAW4V3R5QL3D25UKAA) context deadline exceeded; piecestore: (Node ID: 1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, Piece ID: KUG2JKJTNJWMTZAP5F4HGA2UEKRHORRYVFKQ2FAX4KLQDVEX7UYQ) context deadline exceeded; piecestore: (Node ID: 12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, Piece ID: LTQ6DD2KQ55UCWIJ43SWJO7Q5IDDJFTSD2Z7SVMHR2HLUOUSBNDQ) context deadline exceeded ======LOGS====== ent. Inline: false Jan 13 09:28:06.315478350 (929h59m6.186561385s) [upload=813972]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 13 09:28:06.315483540 (929h59m6.186566575s) [upload=813972]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 13 09:28:09.241621500 (929h59m9.112704545s) [seg_pos={0 0}, upload=813972]: Begin upload segment... Jan 13 09:28:09.241671425 (929h59m9.11275446s) [seg_pos={0 0}, upload=813972]: Done begin upload segment. Jan 13 09:28:09.241703947 (929h59m9.112786992s) [noise=true , address=145.79.227.47:4267 , piece_id=RB3RDZAJRF3VV6LL6LUI5544BM7MFTGUIQ64Y4G23FHFRIYWZNUA, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.241712664 (929h59m9.112795699s) [noise=true , address=145.79.227.49:4337 , piece_id=ZRIX2M4OOA6LKJ6FJ2JORHE4XZQDJRG3KSBUNRJ3EEEXMIK5VMYA, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.242438585 (929h59m9.11352162s) [noise=true , address=145.79.227.2:4113 , piece_id=KUG2JKJTNJWMTZAP5F4HGA2UEKRHORRYVFKQ2FAX4KLQDVEX7UYQ, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.242727556 (929h59m9.113810591s) [noise=true , address=145.79.227.35:4483 , piece_id=PH7UUSSVGW3I2KXRBDWEIFHDL3SC63SYDUPWMKG3NX3HKG6E3OYA, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.243047555 (929h59m9.1141306s) [noise=true , address=145.79.227.5:4221 , piece_id=MAYWAFPWGGY6OJMP2OPGXJ4CFWMHIOL32IDBYGFVIONN6PFRDK2Q, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.243340904 (929h59m9.114423939s) [noise=true , address=145.79.227.48:4301 , piece_id=BIBJBBTITB6AA2SWXXZAIEOOAIARHUIVCBXHB4JXMUSELMZSLLTA, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.243735925 (929h59m9.11481897s) [noise=true , address=145.79.227.37:4566 , piece_id=NHE62LRK3ZLPFLQXOIPJZ3LB47RSNU7L2C3DGJKZYV3EXLYBSN6Q, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.244030276 (929h59m9.115113312s) [noise=true , address=145.79.227.11:4732 , piece_id=ZNFPDXJOGVKRQHPN66JHECSZLNEQ57MVWX4GAA6ETQSCNWCJQ75Q, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.245863306 (929h59m9.116946341s) [noise=true , address=145.79.227.7:4293 , piece_id=LTQ6DD2KQ55UCWIJ43SWJO7Q5IDDJFTSD2Z7SVMHR2HLUOUSBNDQ, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.247027153 (929h59m9.118110188s) [noise=true , address=145.79.227.6:4257 , piece_id=7NPEYE7VHKUHSOYCDOUVMN2HBL4JEXAGZS64K7M6L6DDIIF755TQ, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.248440564 (929h59m9.119523619s) [noise=true , address=145.79.227.31:4799 , piece_id=NOBWZTUK4VDKXM3PCO3Y623KIW2XKFRZXWZEDFVVRT5JHTPCZZQQ, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.249844076 (929h59m9.120927121s) [noise=true , address=145.79.227.50:4373 , piece_id=R5A7RCLHAH3ODMP563M27O6T7ZNACL36HJJOFVLRHYG456XUXXBA, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.251275041 (929h59m9.122358076s) [noise=true , address=145.79.227.52:4447 , piece_id=ZZMECYYJKEXTVZI6WQVEMT23OAVZ7SILIHSSE2OHFKSESUAQHJRQ, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.252655870 (929h59m9.123738915s) [noise=true , address=145.79.227.21:4473 , piece_id=FHOGGYMZY7DU6T7FUFNALWHI4Y6SDMPYF242QLW7C4UZIKKXEJ3A, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.253473537 (929h59m9.124556582s) [noise=true , address=145.79.227.13:4803 , piece_id=DNNAZPZ2TYQI6VCXNIZKSMU5VQCXKEDN3L6K6MHSVNRN3R246BCQ, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.254758193 (929h59m9.125841238s) [noise=true , address=145.79.227.51:4409 , piece_id=3FOIHL6K6JAEDPWIRHT7IJOQ534UUKYAVCNAW4V3R5QL3D25UKAA, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:11.087868620 (929h59m10.958951665s) [upload=813972]: Next returned nil segment Jan 13 09:28:11.087884400 (929h59m10.958967435s) [upload=813972]: Waiting for error group managing segments... Jan 13 09:28:11.388729734 (929h59m11.259812799s) [noise=true , address=145.79.227.50:4373 , piece_id=R5A7RCLHAH3ODMP563M27O6T7ZNACL36HJJOFVLRHYG456XUXXBA, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.399388468 (929h59m11.270471504s) [noise=true , address=145.79.227.21:4473 , piece_id=FHOGGYMZY7DU6T7FUFNALWHI4Y6SDMPYF242QLW7C4UZIKKXEJ3A, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.405180828 (929h59m11.276263873s) [noise=true , address=145.79.227.13:4803 , piece_id=DNNAZPZ2TYQI6VCXNIZKSMU5VQCXKEDN3L6K6MHSVNRN3R246BCQ, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.423915039 (929h59m11.294998074s) [noise=true , address=145.79.227.31:4799 , piece_id=NOBWZTUK4VDKXM3PCO3Y623KIW2XKFRZXWZEDFVVRT5JHTPCZZQQ, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.425205065 (929h59m11.29628811s) [noise=true , address=145.79.227.48:4301 , piece_id=BIBJBBTITB6AA2SWXXZAIEOOAIARHUIVCBXHB4JXMUSELMZSLLTA, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.427897582 (929h59m11.298980617s) [noise=true , address=145.79.227.5:4221 , piece_id=MAYWAFPWGGY6OJMP2OPGXJ4CFWMHIOL32IDBYGFVIONN6PFRDK2Q, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.461421214 (929h59m11.332504249s) [noise=true , address=145.79.227.2:4113 , piece_id=KUG2JKJTNJWMTZAP5F4HGA2UEKRHORRYVFKQ2FAX4KLQDVEX7UYQ, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.466681119 (929h59m11.337764154s) [noise=true , address=145.79.227.37:4566 , piece_id=NHE62LRK3ZLPFLQXOIPJZ3LB47RSNU7L2C3DGJKZYV3EXLYBSN6Q, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.468074292 (929h59m11.339157327s) [noise=true , address=145.79.227.11:4732 , piece_id=ZNFPDXJOGVKRQHPN66JHECSZLNEQ57MVWX4GAA6ETQSCNWCJQ75Q, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.472717633 (929h59m11.343800668s) [noise=true , address=145.79.227.35:4483 , piece_id=PH7UUSSVGW3I2KXRBDWEIFHDL3SC63SYDUPWMKG3NX3HKG6E3OYA, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.502968303 (929h59m11.374051338s) [noise=true , address=145.79.227.7:4293 , piece_id=LTQ6DD2KQ55UCWIJ43SWJO7Q5IDDJFTSD2Z7SVMHR2HLUOUSBNDQ, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.554022174 (929h59m11.425105219s) [noise=true , address=145.79.227.51:4409 , piece_id=3FOIHL6K6JAEDPWIRHT7IJOQ534UUKYAVCNAW4V3R5QL3D25UKAA, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.585092324 (929h59m11.456175369s) [noise=true , address=145.79.227.47:4267 , piece_id=RB3RDZAJRF3VV6LL6LUI5544BM7MFTGUIQ64Y4G23FHFRIYWZNUA, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.585114155 (929h59m11.4561972s) [seg_pos={0 0}, upload=813972]: Segment reached optimal threshold of 13 pieces. Jan 13 09:28:11.585293828 (929h59m11.456376863s) [noise=true , address=145.79.227.52:4447 , piece_id=ZZMECYYJKEXTVZI6WQVEMT23OAVZ7SILIHSSE2OHFKSESUAQHJRQ, seg_pos={0 0}, upload=813972]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP): protocol: write tcp 188.166.225.170:44548->145.79.227.52:4447: use of closed network connection; write tcp 188.166.225.170:44548->145.79.227.52:4447: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44548->145.79.227.52:4447: use of closed network connection Jan 13 09:28:11.585299989 (929h59m11.456383025s) [noise=true , address=145.79.227.52:4447 , piece_id=ZZMECYYJKEXTVZI6WQVEMT23OAVZ7SILIHSSE2OHFKSESUAQHJRQ, seg_pos={0 0}, upload=813972]: Long tail cancellation due to optimal threshold Jan 13 09:28:11.585433102 (929h59m11.456516137s) [noise=true , address=145.79.227.6:4257 , piece_id=7NPEYE7VHKUHSOYCDOUVMN2HBL4JEXAGZS64K7M6L6DDIIF755TQ, seg_pos={0 0}, upload=813972]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK): protocol: write tcp 188.166.225.170:52450->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:52450->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52450->145.79.227.6:4257: use of closed network connection Jan 13 09:28:11.585440497 (929h59m11.456523532s) [noise=true , address=145.79.227.6:4257 , piece_id=7NPEYE7VHKUHSOYCDOUVMN2HBL4JEXAGZS64K7M6L6DDIIF755TQ, seg_pos={0 0}, upload=813972]: Long tail cancellation due to optimal threshold Jan 13 09:28:11.585617424 (929h59m11.456700469s) [noise=true , address=145.79.227.49:4337 , piece_id=ZRIX2M4OOA6LKJ6FJ2JORHE4XZQDJRG3KSBUNRJ3EEEXMIK5VMYA, seg_pos={0 0}, upload=813972]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: read tcp 188.166.225.170:49630->145.79.227.49:4337: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 13 09:28:11.585629516 (929h59m11.456712562s) [noise=true , address=145.79.227.49:4337 , piece_id=ZRIX2M4OOA6LKJ6FJ2JORHE4XZQDJRG3KSBUNRJ3EEEXMIK5VMYA, seg_pos={0 0}, upload=813972]: Long tail cancellation due to optimal threshold Jan 13 09:28:11.585663561 (929h59m11.456746596s) [seg_pos={0 0}, upload=813972]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 13 09:28:11.585783911 (929h59m11.456866946s) [upload=813972]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 13 09:28:11.585791646 (929h59m11.456874681s) [upload=813972]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 13 09:28:11.889773771 (929h59m11.760856816s) [upload=813972]: Done uploading segments... Jan 13 09:28:11.889820049 (929h59m11.760903094s) [upload=813972]: Upload finished. err: <nil> Duration: 5.665785154s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 6.314070575s trace: https://jaeger.dev.storj.io/trace/4226536f67e7860 FAIL (2026-01-13 09:58:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.314097005s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 58 out of 60 is failed (threshold=6) [X.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXX] duration: 5.001603996s trace: https://jaeger.dev.storj.io/trace/7da4c400d5020059 FAIL (2026-01-13 09:58:03 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001647297s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 58 out of 60 is failed (threshold=6) [XX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXX] duration: 5.001395003s trace: https://jaeger.dev.storj.io/trace/6e978ca16233cdf6 FAIL (2026-01-13 09:57:47 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001421151s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [....................X....................X.......X..X....XX.] duration: 27.465658395s trace: https://jaeger.dev.storj.io/trace/1422bf86419b0b28 FAIL (2026-01-13 09:57:44 UTC) orbiter warning: context deadline exceeded Duration: 27.465670779s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 5.686851843s trace: https://jaeger.dev.storj.io/trace/209f6f5543c7dce4 FAIL (2026-01-13 09:57:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: NOBWZTUK4VDKXM3PCO3Y623KIW2XKFRZXWZEDFVVRT5JHTPCZZQQ) context deadline exceeded; piecestore: (Node ID: 12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj, Piece ID: 3FOIHL6K6JAEDPWIRHT7IJOQ534UUKYAVCNAW4V3R5QL3D25UKAA) context deadline exceeded; piecestore: (Node ID: 1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, Piece ID: KUG2JKJTNJWMTZAP5F4HGA2UEKRHORRYVFKQ2FAX4KLQDVEX7UYQ) context deadline exceeded; piecestore: (Node ID: 12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, Piece ID: LTQ6DD2KQ55UCWIJ43SWJO7Q5IDDJFTSD2Z7SVMHR2HLUOUSBNDQ) context deadline exceeded ======LOGS====== ent. Inline: false Jan 13 09:28:06.315478350 (929h59m6.186561385s) [upload=813972]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 13 09:28:06.315483540 (929h59m6.186566575s) [upload=813972]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 13 09:28:09.241621500 (929h59m9.112704545s) [seg_pos={0 0}, upload=813972]: Begin upload segment... Jan 13 09:28:09.241671425 (929h59m9.11275446s) [seg_pos={0 0}, upload=813972]: Done begin upload segment. Jan 13 09:28:09.241703947 (929h59m9.112786992s) [noise=true , address=145.79.227.47:4267 , piece_id=RB3RDZAJRF3VV6LL6LUI5544BM7MFTGUIQ64Y4G23FHFRIYWZNUA, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.241712664 (929h59m9.112795699s) [noise=true , address=145.79.227.49:4337 , piece_id=ZRIX2M4OOA6LKJ6FJ2JORHE4XZQDJRG3KSBUNRJ3EEEXMIK5VMYA, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.242438585 (929h59m9.11352162s) [noise=true , address=145.79.227.2:4113 , piece_id=KUG2JKJTNJWMTZAP5F4HGA2UEKRHORRYVFKQ2FAX4KLQDVEX7UYQ, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.242727556 (929h59m9.113810591s) [noise=true , address=145.79.227.35:4483 , piece_id=PH7UUSSVGW3I2KXRBDWEIFHDL3SC63SYDUPWMKG3NX3HKG6E3OYA, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.243047555 (929h59m9.1141306s) [noise=true , address=145.79.227.5:4221 , piece_id=MAYWAFPWGGY6OJMP2OPGXJ4CFWMHIOL32IDBYGFVIONN6PFRDK2Q, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.243340904 (929h59m9.114423939s) [noise=true , address=145.79.227.48:4301 , piece_id=BIBJBBTITB6AA2SWXXZAIEOOAIARHUIVCBXHB4JXMUSELMZSLLTA, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.243735925 (929h59m9.11481897s) [noise=true , address=145.79.227.37:4566 , piece_id=NHE62LRK3ZLPFLQXOIPJZ3LB47RSNU7L2C3DGJKZYV3EXLYBSN6Q, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.244030276 (929h59m9.115113312s) [noise=true , address=145.79.227.11:4732 , piece_id=ZNFPDXJOGVKRQHPN66JHECSZLNEQ57MVWX4GAA6ETQSCNWCJQ75Q, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.245863306 (929h59m9.116946341s) [noise=true , address=145.79.227.7:4293 , piece_id=LTQ6DD2KQ55UCWIJ43SWJO7Q5IDDJFTSD2Z7SVMHR2HLUOUSBNDQ, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.247027153 (929h59m9.118110188s) [noise=true , address=145.79.227.6:4257 , piece_id=7NPEYE7VHKUHSOYCDOUVMN2HBL4JEXAGZS64K7M6L6DDIIF755TQ, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.248440564 (929h59m9.119523619s) [noise=true , address=145.79.227.31:4799 , piece_id=NOBWZTUK4VDKXM3PCO3Y623KIW2XKFRZXWZEDFVVRT5JHTPCZZQQ, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.249844076 (929h59m9.120927121s) [noise=true , address=145.79.227.50:4373 , piece_id=R5A7RCLHAH3ODMP563M27O6T7ZNACL36HJJOFVLRHYG456XUXXBA, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.251275041 (929h59m9.122358076s) [noise=true , address=145.79.227.52:4447 , piece_id=ZZMECYYJKEXTVZI6WQVEMT23OAVZ7SILIHSSE2OHFKSESUAQHJRQ, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.252655870 (929h59m9.123738915s) [noise=true , address=145.79.227.21:4473 , piece_id=FHOGGYMZY7DU6T7FUFNALWHI4Y6SDMPYF242QLW7C4UZIKKXEJ3A, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.253473537 (929h59m9.124556582s) [noise=true , address=145.79.227.13:4803 , piece_id=DNNAZPZ2TYQI6VCXNIZKSMU5VQCXKEDN3L6K6MHSVNRN3R246BCQ, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.254758193 (929h59m9.125841238s) [noise=true , address=145.79.227.51:4409 , piece_id=3FOIHL6K6JAEDPWIRHT7IJOQ534UUKYAVCNAW4V3R5QL3D25UKAA, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:11.087868620 (929h59m10.958951665s) [upload=813972]: Next returned nil segment Jan 13 09:28:11.087884400 (929h59m10.958967435s) [upload=813972]: Waiting for error group managing segments... Jan 13 09:28:11.388729734 (929h59m11.259812799s) [noise=true , address=145.79.227.50:4373 , piece_id=R5A7RCLHAH3ODMP563M27O6T7ZNACL36HJJOFVLRHYG456XUXXBA, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.399388468 (929h59m11.270471504s) [noise=true , address=145.79.227.21:4473 , piece_id=FHOGGYMZY7DU6T7FUFNALWHI4Y6SDMPYF242QLW7C4UZIKKXEJ3A, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.405180828 (929h59m11.276263873s) [noise=true , address=145.79.227.13:4803 , piece_id=DNNAZPZ2TYQI6VCXNIZKSMU5VQCXKEDN3L6K6MHSVNRN3R246BCQ, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.423915039 (929h59m11.294998074s) [noise=true , address=145.79.227.31:4799 , piece_id=NOBWZTUK4VDKXM3PCO3Y623KIW2XKFRZXWZEDFVVRT5JHTPCZZQQ, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.425205065 (929h59m11.29628811s) [noise=true , address=145.79.227.48:4301 , piece_id=BIBJBBTITB6AA2SWXXZAIEOOAIARHUIVCBXHB4JXMUSELMZSLLTA, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.427897582 (929h59m11.298980617s) [noise=true , address=145.79.227.5:4221 , piece_id=MAYWAFPWGGY6OJMP2OPGXJ4CFWMHIOL32IDBYGFVIONN6PFRDK2Q, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.461421214 (929h59m11.332504249s) [noise=true , address=145.79.227.2:4113 , piece_id=KUG2JKJTNJWMTZAP5F4HGA2UEKRHORRYVFKQ2FAX4KLQDVEX7UYQ, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.466681119 (929h59m11.337764154s) [noise=true , address=145.79.227.37:4566 , piece_id=NHE62LRK3ZLPFLQXOIPJZ3LB47RSNU7L2C3DGJKZYV3EXLYBSN6Q, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.468074292 (929h59m11.339157327s) [noise=true , address=145.79.227.11:4732 , piece_id=ZNFPDXJOGVKRQHPN66JHECSZLNEQ57MVWX4GAA6ETQSCNWCJQ75Q, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.472717633 (929h59m11.343800668s) [noise=true , address=145.79.227.35:4483 , piece_id=PH7UUSSVGW3I2KXRBDWEIFHDL3SC63SYDUPWMKG3NX3HKG6E3OYA, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.502968303 (929h59m11.374051338s) [noise=true , address=145.79.227.7:4293 , piece_id=LTQ6DD2KQ55UCWIJ43SWJO7Q5IDDJFTSD2Z7SVMHR2HLUOUSBNDQ, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.554022174 (929h59m11.425105219s) [noise=true , address=145.79.227.51:4409 , piece_id=3FOIHL6K6JAEDPWIRHT7IJOQ534UUKYAVCNAW4V3R5QL3D25UKAA, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.585092324 (929h59m11.456175369s) [noise=true , address=145.79.227.47:4267 , piece_id=RB3RDZAJRF3VV6LL6LUI5544BM7MFTGUIQ64Y4G23FHFRIYWZNUA, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.585114155 (929h59m11.4561972s) [seg_pos={0 0}, upload=813972]: Segment reached optimal threshold of 13 pieces. Jan 13 09:28:11.585293828 (929h59m11.456376863s) [noise=true , address=145.79.227.52:4447 , piece_id=ZZMECYYJKEXTVZI6WQVEMT23OAVZ7SILIHSSE2OHFKSESUAQHJRQ, seg_pos={0 0}, upload=813972]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP): protocol: write tcp 188.166.225.170:44548->145.79.227.52:4447: use of closed network connection; write tcp 188.166.225.170:44548->145.79.227.52:4447: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44548->145.79.227.52:4447: use of closed network connection Jan 13 09:28:11.585299989 (929h59m11.456383025s) [noise=true , address=145.79.227.52:4447 , piece_id=ZZMECYYJKEXTVZI6WQVEMT23OAVZ7SILIHSSE2OHFKSESUAQHJRQ, seg_pos={0 0}, upload=813972]: Long tail cancellation due to optimal threshold Jan 13 09:28:11.585433102 (929h59m11.456516137s) [noise=true , address=145.79.227.6:4257 , piece_id=7NPEYE7VHKUHSOYCDOUVMN2HBL4JEXAGZS64K7M6L6DDIIF755TQ, seg_pos={0 0}, upload=813972]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK): protocol: write tcp 188.166.225.170:52450->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:52450->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52450->145.79.227.6:4257: use of closed network connection Jan 13 09:28:11.585440497 (929h59m11.456523532s) [noise=true , address=145.79.227.6:4257 , piece_id=7NPEYE7VHKUHSOYCDOUVMN2HBL4JEXAGZS64K7M6L6DDIIF755TQ, seg_pos={0 0}, upload=813972]: Long tail cancellation due to optimal threshold Jan 13 09:28:11.585617424 (929h59m11.456700469s) [noise=true , address=145.79.227.49:4337 , piece_id=ZRIX2M4OOA6LKJ6FJ2JORHE4XZQDJRG3KSBUNRJ3EEEXMIK5VMYA, seg_pos={0 0}, upload=813972]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: read tcp 188.166.225.170:49630->145.79.227.49:4337: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 13 09:28:11.585629516 (929h59m11.456712562s) [noise=true , address=145.79.227.49:4337 , piece_id=ZRIX2M4OOA6LKJ6FJ2JORHE4XZQDJRG3KSBUNRJ3EEEXMIK5VMYA, seg_pos={0 0}, upload=813972]: Long tail cancellation due to optimal threshold Jan 13 09:28:11.585663561 (929h59m11.456746596s) [seg_pos={0 0}, upload=813972]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 13 09:28:11.585783911 (929h59m11.456866946s) [upload=813972]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 13 09:28:11.585791646 (929h59m11.456874681s) [upload=813972]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 13 09:28:11.889773771 (929h59m11.760856816s) [upload=813972]: Done uploading segments... Jan 13 09:28:11.889820049 (929h59m11.760903094s) [upload=813972]: Upload finished. err: <nil> Duration: 5.686878584s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....................X....................X.......X..X....XX] duration: 30.001298085s trace: https://jaeger.dev.storj.io/trace/1422bf86419b0b28 FAIL (2026-01-13 09:57:16 UTC) orbiter warning: context deadline exceeded Duration: 30.001320908s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 58 out of 60 is failed (threshold=6) [XXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXX] duration: 5.001516844s trace: https://jaeger.dev.storj.io/trace/14c1b5aeb72472de FAIL (2026-01-13 09:56:58 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001544787s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 8.704224992s trace: https://jaeger.dev.storj.io/trace/4226536f67e7860 FAIL (2026-01-13 09:56:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 8.704252022s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 5.720208646s trace: https://jaeger.dev.storj.io/trace/209f6f5543c7dce4 FAIL (2026-01-13 09:56:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: NOBWZTUK4VDKXM3PCO3Y623KIW2XKFRZXWZEDFVVRT5JHTPCZZQQ) context deadline exceeded; piecestore: (Node ID: 12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj, Piece ID: 3FOIHL6K6JAEDPWIRHT7IJOQ534UUKYAVCNAW4V3R5QL3D25UKAA) context deadline exceeded; piecestore: (Node ID: 1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, Piece ID: KUG2JKJTNJWMTZAP5F4HGA2UEKRHORRYVFKQ2FAX4KLQDVEX7UYQ) context deadline exceeded; piecestore: (Node ID: 12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, Piece ID: LTQ6DD2KQ55UCWIJ43SWJO7Q5IDDJFTSD2Z7SVMHR2HLUOUSBNDQ) context deadline exceeded ======LOGS====== ent. Inline: false Jan 13 09:28:06.315478350 (929h59m6.186561385s) [upload=813972]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 13 09:28:06.315483540 (929h59m6.186566575s) [upload=813972]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 13 09:28:09.241621500 (929h59m9.112704545s) [seg_pos={0 0}, upload=813972]: Begin upload segment... Jan 13 09:28:09.241671425 (929h59m9.11275446s) [seg_pos={0 0}, upload=813972]: Done begin upload segment. Jan 13 09:28:09.241703947 (929h59m9.112786992s) [noise=true , address=145.79.227.47:4267 , piece_id=RB3RDZAJRF3VV6LL6LUI5544BM7MFTGUIQ64Y4G23FHFRIYWZNUA, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.241712664 (929h59m9.112795699s) [noise=true , address=145.79.227.49:4337 , piece_id=ZRIX2M4OOA6LKJ6FJ2JORHE4XZQDJRG3KSBUNRJ3EEEXMIK5VMYA, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.242438585 (929h59m9.11352162s) [noise=true , address=145.79.227.2:4113 , piece_id=KUG2JKJTNJWMTZAP5F4HGA2UEKRHORRYVFKQ2FAX4KLQDVEX7UYQ, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.242727556 (929h59m9.113810591s) [noise=true , address=145.79.227.35:4483 , piece_id=PH7UUSSVGW3I2KXRBDWEIFHDL3SC63SYDUPWMKG3NX3HKG6E3OYA, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.243047555 (929h59m9.1141306s) [noise=true , address=145.79.227.5:4221 , piece_id=MAYWAFPWGGY6OJMP2OPGXJ4CFWMHIOL32IDBYGFVIONN6PFRDK2Q, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.243340904 (929h59m9.114423939s) [noise=true , address=145.79.227.48:4301 , piece_id=BIBJBBTITB6AA2SWXXZAIEOOAIARHUIVCBXHB4JXMUSELMZSLLTA, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.243735925 (929h59m9.11481897s) [noise=true , address=145.79.227.37:4566 , piece_id=NHE62LRK3ZLPFLQXOIPJZ3LB47RSNU7L2C3DGJKZYV3EXLYBSN6Q, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.244030276 (929h59m9.115113312s) [noise=true , address=145.79.227.11:4732 , piece_id=ZNFPDXJOGVKRQHPN66JHECSZLNEQ57MVWX4GAA6ETQSCNWCJQ75Q, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.245863306 (929h59m9.116946341s) [noise=true , address=145.79.227.7:4293 , piece_id=LTQ6DD2KQ55UCWIJ43SWJO7Q5IDDJFTSD2Z7SVMHR2HLUOUSBNDQ, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.247027153 (929h59m9.118110188s) [noise=true , address=145.79.227.6:4257 , piece_id=7NPEYE7VHKUHSOYCDOUVMN2HBL4JEXAGZS64K7M6L6DDIIF755TQ, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.248440564 (929h59m9.119523619s) [noise=true , address=145.79.227.31:4799 , piece_id=NOBWZTUK4VDKXM3PCO3Y623KIW2XKFRZXWZEDFVVRT5JHTPCZZQQ, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.249844076 (929h59m9.120927121s) [noise=true , address=145.79.227.50:4373 , piece_id=R5A7RCLHAH3ODMP563M27O6T7ZNACL36HJJOFVLRHYG456XUXXBA, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.251275041 (929h59m9.122358076s) [noise=true , address=145.79.227.52:4447 , piece_id=ZZMECYYJKEXTVZI6WQVEMT23OAVZ7SILIHSSE2OHFKSESUAQHJRQ, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.252655870 (929h59m9.123738915s) [noise=true , address=145.79.227.21:4473 , piece_id=FHOGGYMZY7DU6T7FUFNALWHI4Y6SDMPYF242QLW7C4UZIKKXEJ3A, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.253473537 (929h59m9.124556582s) [noise=true , address=145.79.227.13:4803 , piece_id=DNNAZPZ2TYQI6VCXNIZKSMU5VQCXKEDN3L6K6MHSVNRN3R246BCQ, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.254758193 (929h59m9.125841238s) [noise=true , address=145.79.227.51:4409 , piece_id=3FOIHL6K6JAEDPWIRHT7IJOQ534UUKYAVCNAW4V3R5QL3D25UKAA, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:11.087868620 (929h59m10.958951665s) [upload=813972]: Next returned nil segment Jan 13 09:28:11.087884400 (929h59m10.958967435s) [upload=813972]: Waiting for error group managing segments... Jan 13 09:28:11.388729734 (929h59m11.259812799s) [noise=true , address=145.79.227.50:4373 , piece_id=R5A7RCLHAH3ODMP563M27O6T7ZNACL36HJJOFVLRHYG456XUXXBA, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.399388468 (929h59m11.270471504s) [noise=true , address=145.79.227.21:4473 , piece_id=FHOGGYMZY7DU6T7FUFNALWHI4Y6SDMPYF242QLW7C4UZIKKXEJ3A, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.405180828 (929h59m11.276263873s) [noise=true , address=145.79.227.13:4803 , piece_id=DNNAZPZ2TYQI6VCXNIZKSMU5VQCXKEDN3L6K6MHSVNRN3R246BCQ, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.423915039 (929h59m11.294998074s) [noise=true , address=145.79.227.31:4799 , piece_id=NOBWZTUK4VDKXM3PCO3Y623KIW2XKFRZXWZEDFVVRT5JHTPCZZQQ, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.425205065 (929h59m11.29628811s) [noise=true , address=145.79.227.48:4301 , piece_id=BIBJBBTITB6AA2SWXXZAIEOOAIARHUIVCBXHB4JXMUSELMZSLLTA, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.427897582 (929h59m11.298980617s) [noise=true , address=145.79.227.5:4221 , piece_id=MAYWAFPWGGY6OJMP2OPGXJ4CFWMHIOL32IDBYGFVIONN6PFRDK2Q, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.461421214 (929h59m11.332504249s) [noise=true , address=145.79.227.2:4113 , piece_id=KUG2JKJTNJWMTZAP5F4HGA2UEKRHORRYVFKQ2FAX4KLQDVEX7UYQ, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.466681119 (929h59m11.337764154s) [noise=true , address=145.79.227.37:4566 , piece_id=NHE62LRK3ZLPFLQXOIPJZ3LB47RSNU7L2C3DGJKZYV3EXLYBSN6Q, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.468074292 (929h59m11.339157327s) [noise=true , address=145.79.227.11:4732 , piece_id=ZNFPDXJOGVKRQHPN66JHECSZLNEQ57MVWX4GAA6ETQSCNWCJQ75Q, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.472717633 (929h59m11.343800668s) [noise=true , address=145.79.227.35:4483 , piece_id=PH7UUSSVGW3I2KXRBDWEIFHDL3SC63SYDUPWMKG3NX3HKG6E3OYA, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.502968303 (929h59m11.374051338s) [noise=true , address=145.79.227.7:4293 , piece_id=LTQ6DD2KQ55UCWIJ43SWJO7Q5IDDJFTSD2Z7SVMHR2HLUOUSBNDQ, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.554022174 (929h59m11.425105219s) [noise=true , address=145.79.227.51:4409 , piece_id=3FOIHL6K6JAEDPWIRHT7IJOQ534UUKYAVCNAW4V3R5QL3D25UKAA, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.585092324 (929h59m11.456175369s) [noise=true , address=145.79.227.47:4267 , piece_id=RB3RDZAJRF3VV6LL6LUI5544BM7MFTGUIQ64Y4G23FHFRIYWZNUA, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.585114155 (929h59m11.4561972s) [seg_pos={0 0}, upload=813972]: Segment reached optimal threshold of 13 pieces. Jan 13 09:28:11.585293828 (929h59m11.456376863s) [noise=true , address=145.79.227.52:4447 , piece_id=ZZMECYYJKEXTVZI6WQVEMT23OAVZ7SILIHSSE2OHFKSESUAQHJRQ, seg_pos={0 0}, upload=813972]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP): protocol: write tcp 188.166.225.170:44548->145.79.227.52:4447: use of closed network connection; write tcp 188.166.225.170:44548->145.79.227.52:4447: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44548->145.79.227.52:4447: use of closed network connection Jan 13 09:28:11.585299989 (929h59m11.456383025s) [noise=true , address=145.79.227.52:4447 , piece_id=ZZMECYYJKEXTVZI6WQVEMT23OAVZ7SILIHSSE2OHFKSESUAQHJRQ, seg_pos={0 0}, upload=813972]: Long tail cancellation due to optimal threshold Jan 13 09:28:11.585433102 (929h59m11.456516137s) [noise=true , address=145.79.227.6:4257 , piece_id=7NPEYE7VHKUHSOYCDOUVMN2HBL4JEXAGZS64K7M6L6DDIIF755TQ, seg_pos={0 0}, upload=813972]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK): protocol: write tcp 188.166.225.170:52450->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:52450->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52450->145.79.227.6:4257: use of closed network connection Jan 13 09:28:11.585440497 (929h59m11.456523532s) [noise=true , address=145.79.227.6:4257 , piece_id=7NPEYE7VHKUHSOYCDOUVMN2HBL4JEXAGZS64K7M6L6DDIIF755TQ, seg_pos={0 0}, upload=813972]: Long tail cancellation due to optimal threshold Jan 13 09:28:11.585617424 (929h59m11.456700469s) [noise=true , address=145.79.227.49:4337 , piece_id=ZRIX2M4OOA6LKJ6FJ2JORHE4XZQDJRG3KSBUNRJ3EEEXMIK5VMYA, seg_pos={0 0}, upload=813972]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: read tcp 188.166.225.170:49630->145.79.227.49:4337: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 13 09:28:11.585629516 (929h59m11.456712562s) [noise=true , address=145.79.227.49:4337 , piece_id=ZRIX2M4OOA6LKJ6FJ2JORHE4XZQDJRG3KSBUNRJ3EEEXMIK5VMYA, seg_pos={0 0}, upload=813972]: Long tail cancellation due to optimal threshold Jan 13 09:28:11.585663561 (929h59m11.456746596s) [seg_pos={0 0}, upload=813972]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 13 09:28:11.585783911 (929h59m11.456866946s) [upload=813972]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 13 09:28:11.585791646 (929h59m11.456874681s) [upload=813972]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 13 09:28:11.889773771 (929h59m11.760856816s) [upload=813972]: Done uploading segments... Jan 13 09:28:11.889820049 (929h59m11.760903094s) [upload=813972]: Upload finished. err: <nil> Duration: 5.720238703s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 58 out of 60 is failed (threshold=6) [XXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXX] duration: 5.001641733s trace: https://jaeger.dev.storj.io/trace/7a9fb135535dcb48 FAIL (2026-01-13 09:56:18 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001663464s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 6.39508358s trace: https://jaeger.dev.storj.io/trace/4226536f67e7860 FAIL (2026-01-13 09:56:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.395106514s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [......................X....................X.......X..X....X] duration: 30.00119053s trace: https://jaeger.dev.storj.io/trace/4ba9dc9b6ba11e12 FAIL (2026-01-13 09:56:06 UTC) orbiter warning: context deadline exceeded Duration: 30.001213072s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 5.93351417s trace: https://jaeger.dev.storj.io/trace/209f6f5543c7dce4 FAIL (2026-01-13 09:55:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: NOBWZTUK4VDKXM3PCO3Y623KIW2XKFRZXWZEDFVVRT5JHTPCZZQQ) context deadline exceeded; piecestore: (Node ID: 12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj, Piece ID: 3FOIHL6K6JAEDPWIRHT7IJOQ534UUKYAVCNAW4V3R5QL3D25UKAA) context deadline exceeded; piecestore: (Node ID: 1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, Piece ID: KUG2JKJTNJWMTZAP5F4HGA2UEKRHORRYVFKQ2FAX4KLQDVEX7UYQ) context deadline exceeded; piecestore: (Node ID: 12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, Piece ID: LTQ6DD2KQ55UCWIJ43SWJO7Q5IDDJFTSD2Z7SVMHR2HLUOUSBNDQ) context deadline exceeded ======LOGS====== ent. Inline: false Jan 13 09:28:06.315478350 (929h59m6.186561385s) [upload=813972]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 13 09:28:06.315483540 (929h59m6.186566575s) [upload=813972]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 13 09:28:09.241621500 (929h59m9.112704545s) [seg_pos={0 0}, upload=813972]: Begin upload segment... Jan 13 09:28:09.241671425 (929h59m9.11275446s) [seg_pos={0 0}, upload=813972]: Done begin upload segment. Jan 13 09:28:09.241703947 (929h59m9.112786992s) [noise=true , address=145.79.227.47:4267 , piece_id=RB3RDZAJRF3VV6LL6LUI5544BM7MFTGUIQ64Y4G23FHFRIYWZNUA, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.241712664 (929h59m9.112795699s) [noise=true , address=145.79.227.49:4337 , piece_id=ZRIX2M4OOA6LKJ6FJ2JORHE4XZQDJRG3KSBUNRJ3EEEXMIK5VMYA, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.242438585 (929h59m9.11352162s) [noise=true , address=145.79.227.2:4113 , piece_id=KUG2JKJTNJWMTZAP5F4HGA2UEKRHORRYVFKQ2FAX4KLQDVEX7UYQ, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.242727556 (929h59m9.113810591s) [noise=true , address=145.79.227.35:4483 , piece_id=PH7UUSSVGW3I2KXRBDWEIFHDL3SC63SYDUPWMKG3NX3HKG6E3OYA, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.243047555 (929h59m9.1141306s) [noise=true , address=145.79.227.5:4221 , piece_id=MAYWAFPWGGY6OJMP2OPGXJ4CFWMHIOL32IDBYGFVIONN6PFRDK2Q, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.243340904 (929h59m9.114423939s) [noise=true , address=145.79.227.48:4301 , piece_id=BIBJBBTITB6AA2SWXXZAIEOOAIARHUIVCBXHB4JXMUSELMZSLLTA, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.243735925 (929h59m9.11481897s) [noise=true , address=145.79.227.37:4566 , piece_id=NHE62LRK3ZLPFLQXOIPJZ3LB47RSNU7L2C3DGJKZYV3EXLYBSN6Q, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.244030276 (929h59m9.115113312s) [noise=true , address=145.79.227.11:4732 , piece_id=ZNFPDXJOGVKRQHPN66JHECSZLNEQ57MVWX4GAA6ETQSCNWCJQ75Q, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.245863306 (929h59m9.116946341s) [noise=true , address=145.79.227.7:4293 , piece_id=LTQ6DD2KQ55UCWIJ43SWJO7Q5IDDJFTSD2Z7SVMHR2HLUOUSBNDQ, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.247027153 (929h59m9.118110188s) [noise=true , address=145.79.227.6:4257 , piece_id=7NPEYE7VHKUHSOYCDOUVMN2HBL4JEXAGZS64K7M6L6DDIIF755TQ, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.248440564 (929h59m9.119523619s) [noise=true , address=145.79.227.31:4799 , piece_id=NOBWZTUK4VDKXM3PCO3Y623KIW2XKFRZXWZEDFVVRT5JHTPCZZQQ, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.249844076 (929h59m9.120927121s) [noise=true , address=145.79.227.50:4373 , piece_id=R5A7RCLHAH3ODMP563M27O6T7ZNACL36HJJOFVLRHYG456XUXXBA, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.251275041 (929h59m9.122358076s) [noise=true , address=145.79.227.52:4447 , piece_id=ZZMECYYJKEXTVZI6WQVEMT23OAVZ7SILIHSSE2OHFKSESUAQHJRQ, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.252655870 (929h59m9.123738915s) [noise=true , address=145.79.227.21:4473 , piece_id=FHOGGYMZY7DU6T7FUFNALWHI4Y6SDMPYF242QLW7C4UZIKKXEJ3A, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.253473537 (929h59m9.124556582s) [noise=true , address=145.79.227.13:4803 , piece_id=DNNAZPZ2TYQI6VCXNIZKSMU5VQCXKEDN3L6K6MHSVNRN3R246BCQ, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.254758193 (929h59m9.125841238s) [noise=true , address=145.79.227.51:4409 , piece_id=3FOIHL6K6JAEDPWIRHT7IJOQ534UUKYAVCNAW4V3R5QL3D25UKAA, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:11.087868620 (929h59m10.958951665s) [upload=813972]: Next returned nil segment Jan 13 09:28:11.087884400 (929h59m10.958967435s) [upload=813972]: Waiting for error group managing segments... Jan 13 09:28:11.388729734 (929h59m11.259812799s) [noise=true , address=145.79.227.50:4373 , piece_id=R5A7RCLHAH3ODMP563M27O6T7ZNACL36HJJOFVLRHYG456XUXXBA, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.399388468 (929h59m11.270471504s) [noise=true , address=145.79.227.21:4473 , piece_id=FHOGGYMZY7DU6T7FUFNALWHI4Y6SDMPYF242QLW7C4UZIKKXEJ3A, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.405180828 (929h59m11.276263873s) [noise=true , address=145.79.227.13:4803 , piece_id=DNNAZPZ2TYQI6VCXNIZKSMU5VQCXKEDN3L6K6MHSVNRN3R246BCQ, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.423915039 (929h59m11.294998074s) [noise=true , address=145.79.227.31:4799 , piece_id=NOBWZTUK4VDKXM3PCO3Y623KIW2XKFRZXWZEDFVVRT5JHTPCZZQQ, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.425205065 (929h59m11.29628811s) [noise=true , address=145.79.227.48:4301 , piece_id=BIBJBBTITB6AA2SWXXZAIEOOAIARHUIVCBXHB4JXMUSELMZSLLTA, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.427897582 (929h59m11.298980617s) [noise=true , address=145.79.227.5:4221 , piece_id=MAYWAFPWGGY6OJMP2OPGXJ4CFWMHIOL32IDBYGFVIONN6PFRDK2Q, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.461421214 (929h59m11.332504249s) [noise=true , address=145.79.227.2:4113 , piece_id=KUG2JKJTNJWMTZAP5F4HGA2UEKRHORRYVFKQ2FAX4KLQDVEX7UYQ, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.466681119 (929h59m11.337764154s) [noise=true , address=145.79.227.37:4566 , piece_id=NHE62LRK3ZLPFLQXOIPJZ3LB47RSNU7L2C3DGJKZYV3EXLYBSN6Q, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.468074292 (929h59m11.339157327s) [noise=true , address=145.79.227.11:4732 , piece_id=ZNFPDXJOGVKRQHPN66JHECSZLNEQ57MVWX4GAA6ETQSCNWCJQ75Q, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.472717633 (929h59m11.343800668s) [noise=true , address=145.79.227.35:4483 , piece_id=PH7UUSSVGW3I2KXRBDWEIFHDL3SC63SYDUPWMKG3NX3HKG6E3OYA, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.502968303 (929h59m11.374051338s) [noise=true , address=145.79.227.7:4293 , piece_id=LTQ6DD2KQ55UCWIJ43SWJO7Q5IDDJFTSD2Z7SVMHR2HLUOUSBNDQ, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.554022174 (929h59m11.425105219s) [noise=true , address=145.79.227.51:4409 , piece_id=3FOIHL6K6JAEDPWIRHT7IJOQ534UUKYAVCNAW4V3R5QL3D25UKAA, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.585092324 (929h59m11.456175369s) [noise=true , address=145.79.227.47:4267 , piece_id=RB3RDZAJRF3VV6LL6LUI5544BM7MFTGUIQ64Y4G23FHFRIYWZNUA, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.585114155 (929h59m11.4561972s) [seg_pos={0 0}, upload=813972]: Segment reached optimal threshold of 13 pieces. Jan 13 09:28:11.585293828 (929h59m11.456376863s) [noise=true , address=145.79.227.52:4447 , piece_id=ZZMECYYJKEXTVZI6WQVEMT23OAVZ7SILIHSSE2OHFKSESUAQHJRQ, seg_pos={0 0}, upload=813972]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP): protocol: write tcp 188.166.225.170:44548->145.79.227.52:4447: use of closed network connection; write tcp 188.166.225.170:44548->145.79.227.52:4447: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44548->145.79.227.52:4447: use of closed network connection Jan 13 09:28:11.585299989 (929h59m11.456383025s) [noise=true , address=145.79.227.52:4447 , piece_id=ZZMECYYJKEXTVZI6WQVEMT23OAVZ7SILIHSSE2OHFKSESUAQHJRQ, seg_pos={0 0}, upload=813972]: Long tail cancellation due to optimal threshold Jan 13 09:28:11.585433102 (929h59m11.456516137s) [noise=true , address=145.79.227.6:4257 , piece_id=7NPEYE7VHKUHSOYCDOUVMN2HBL4JEXAGZS64K7M6L6DDIIF755TQ, seg_pos={0 0}, upload=813972]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK): protocol: write tcp 188.166.225.170:52450->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:52450->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52450->145.79.227.6:4257: use of closed network connection Jan 13 09:28:11.585440497 (929h59m11.456523532s) [noise=true , address=145.79.227.6:4257 , piece_id=7NPEYE7VHKUHSOYCDOUVMN2HBL4JEXAGZS64K7M6L6DDIIF755TQ, seg_pos={0 0}, upload=813972]: Long tail cancellation due to optimal threshold Jan 13 09:28:11.585617424 (929h59m11.456700469s) [noise=true , address=145.79.227.49:4337 , piece_id=ZRIX2M4OOA6LKJ6FJ2JORHE4XZQDJRG3KSBUNRJ3EEEXMIK5VMYA, seg_pos={0 0}, upload=813972]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: read tcp 188.166.225.170:49630->145.79.227.49:4337: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 13 09:28:11.585629516 (929h59m11.456712562s) [noise=true , address=145.79.227.49:4337 , piece_id=ZRIX2M4OOA6LKJ6FJ2JORHE4XZQDJRG3KSBUNRJ3EEEXMIK5VMYA, seg_pos={0 0}, upload=813972]: Long tail cancellation due to optimal threshold Jan 13 09:28:11.585663561 (929h59m11.456746596s) [seg_pos={0 0}, upload=813972]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 13 09:28:11.585783911 (929h59m11.456866946s) [upload=813972]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 13 09:28:11.585791646 (929h59m11.456874681s) [upload=813972]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 13 09:28:11.889773771 (929h59m11.760856816s) [upload=813972]: Done uploading segments... Jan 13 09:28:11.889820049 (929h59m11.760903094s) [upload=813972]: Upload finished. err: <nil> Duration: 5.933534099s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 58 out of 60 is failed (threshold=6) [XXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXX] duration: 5.00095403s trace: https://jaeger.dev.storj.io/trace/22241dad6186dfd4 FAIL (2026-01-13 09:55:38 UTC) orbiter warning: context deadline exceeded Duration: 5.000968346s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 14.902070088s trace: https://jaeger.dev.storj.io/trace/4226536f67e7860 FAIL (2026-01-13 09:55:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 14.902151122s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 58 out of 60 is failed (threshold=6) [XXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXX] duration: 5.001556187s trace: https://jaeger.dev.storj.io/trace/2bb90d7f67906f1 FAIL (2026-01-13 09:54:55 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001572327s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 5.788088314s trace: https://jaeger.dev.storj.io/trace/209f6f5543c7dce4 FAIL (2026-01-13 09:54:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: NOBWZTUK4VDKXM3PCO3Y623KIW2XKFRZXWZEDFVVRT5JHTPCZZQQ) context deadline exceeded; piecestore: (Node ID: 12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj, Piece ID: 3FOIHL6K6JAEDPWIRHT7IJOQ534UUKYAVCNAW4V3R5QL3D25UKAA) context deadline exceeded; piecestore: (Node ID: 1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, Piece ID: KUG2JKJTNJWMTZAP5F4HGA2UEKRHORRYVFKQ2FAX4KLQDVEX7UYQ) context deadline exceeded; piecestore: (Node ID: 12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, Piece ID: LTQ6DD2KQ55UCWIJ43SWJO7Q5IDDJFTSD2Z7SVMHR2HLUOUSBNDQ) context deadline exceeded ======LOGS====== ent. Inline: false Jan 13 09:28:06.315478350 (929h59m6.186561385s) [upload=813972]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 13 09:28:06.315483540 (929h59m6.186566575s) [upload=813972]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 13 09:28:09.241621500 (929h59m9.112704545s) [seg_pos={0 0}, upload=813972]: Begin upload segment... Jan 13 09:28:09.241671425 (929h59m9.11275446s) [seg_pos={0 0}, upload=813972]: Done begin upload segment. Jan 13 09:28:09.241703947 (929h59m9.112786992s) [noise=true , address=145.79.227.47:4267 , piece_id=RB3RDZAJRF3VV6LL6LUI5544BM7MFTGUIQ64Y4G23FHFRIYWZNUA, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.241712664 (929h59m9.112795699s) [noise=true , address=145.79.227.49:4337 , piece_id=ZRIX2M4OOA6LKJ6FJ2JORHE4XZQDJRG3KSBUNRJ3EEEXMIK5VMYA, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.242438585 (929h59m9.11352162s) [noise=true , address=145.79.227.2:4113 , piece_id=KUG2JKJTNJWMTZAP5F4HGA2UEKRHORRYVFKQ2FAX4KLQDVEX7UYQ, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.242727556 (929h59m9.113810591s) [noise=true , address=145.79.227.35:4483 , piece_id=PH7UUSSVGW3I2KXRBDWEIFHDL3SC63SYDUPWMKG3NX3HKG6E3OYA, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.243047555 (929h59m9.1141306s) [noise=true , address=145.79.227.5:4221 , piece_id=MAYWAFPWGGY6OJMP2OPGXJ4CFWMHIOL32IDBYGFVIONN6PFRDK2Q, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.243340904 (929h59m9.114423939s) [noise=true , address=145.79.227.48:4301 , piece_id=BIBJBBTITB6AA2SWXXZAIEOOAIARHUIVCBXHB4JXMUSELMZSLLTA, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.243735925 (929h59m9.11481897s) [noise=true , address=145.79.227.37:4566 , piece_id=NHE62LRK3ZLPFLQXOIPJZ3LB47RSNU7L2C3DGJKZYV3EXLYBSN6Q, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.244030276 (929h59m9.115113312s) [noise=true , address=145.79.227.11:4732 , piece_id=ZNFPDXJOGVKRQHPN66JHECSZLNEQ57MVWX4GAA6ETQSCNWCJQ75Q, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.245863306 (929h59m9.116946341s) [noise=true , address=145.79.227.7:4293 , piece_id=LTQ6DD2KQ55UCWIJ43SWJO7Q5IDDJFTSD2Z7SVMHR2HLUOUSBNDQ, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.247027153 (929h59m9.118110188s) [noise=true , address=145.79.227.6:4257 , piece_id=7NPEYE7VHKUHSOYCDOUVMN2HBL4JEXAGZS64K7M6L6DDIIF755TQ, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.248440564 (929h59m9.119523619s) [noise=true , address=145.79.227.31:4799 , piece_id=NOBWZTUK4VDKXM3PCO3Y623KIW2XKFRZXWZEDFVVRT5JHTPCZZQQ, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.249844076 (929h59m9.120927121s) [noise=true , address=145.79.227.50:4373 , piece_id=R5A7RCLHAH3ODMP563M27O6T7ZNACL36HJJOFVLRHYG456XUXXBA, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.251275041 (929h59m9.122358076s) [noise=true , address=145.79.227.52:4447 , piece_id=ZZMECYYJKEXTVZI6WQVEMT23OAVZ7SILIHSSE2OHFKSESUAQHJRQ, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.252655870 (929h59m9.123738915s) [noise=true , address=145.79.227.21:4473 , piece_id=FHOGGYMZY7DU6T7FUFNALWHI4Y6SDMPYF242QLW7C4UZIKKXEJ3A, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.253473537 (929h59m9.124556582s) [noise=true , address=145.79.227.13:4803 , piece_id=DNNAZPZ2TYQI6VCXNIZKSMU5VQCXKEDN3L6K6MHSVNRN3R246BCQ, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.254758193 (929h59m9.125841238s) [noise=true , address=145.79.227.51:4409 , piece_id=3FOIHL6K6JAEDPWIRHT7IJOQ534UUKYAVCNAW4V3R5QL3D25UKAA, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:11.087868620 (929h59m10.958951665s) [upload=813972]: Next returned nil segment Jan 13 09:28:11.087884400 (929h59m10.958967435s) [upload=813972]: Waiting for error group managing segments... Jan 13 09:28:11.388729734 (929h59m11.259812799s) [noise=true , address=145.79.227.50:4373 , piece_id=R5A7RCLHAH3ODMP563M27O6T7ZNACL36HJJOFVLRHYG456XUXXBA, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.399388468 (929h59m11.270471504s) [noise=true , address=145.79.227.21:4473 , piece_id=FHOGGYMZY7DU6T7FUFNALWHI4Y6SDMPYF242QLW7C4UZIKKXEJ3A, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.405180828 (929h59m11.276263873s) [noise=true , address=145.79.227.13:4803 , piece_id=DNNAZPZ2TYQI6VCXNIZKSMU5VQCXKEDN3L6K6MHSVNRN3R246BCQ, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.423915039 (929h59m11.294998074s) [noise=true , address=145.79.227.31:4799 , piece_id=NOBWZTUK4VDKXM3PCO3Y623KIW2XKFRZXWZEDFVVRT5JHTPCZZQQ, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.425205065 (929h59m11.29628811s) [noise=true , address=145.79.227.48:4301 , piece_id=BIBJBBTITB6AA2SWXXZAIEOOAIARHUIVCBXHB4JXMUSELMZSLLTA, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.427897582 (929h59m11.298980617s) [noise=true , address=145.79.227.5:4221 , piece_id=MAYWAFPWGGY6OJMP2OPGXJ4CFWMHIOL32IDBYGFVIONN6PFRDK2Q, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.461421214 (929h59m11.332504249s) [noise=true , address=145.79.227.2:4113 , piece_id=KUG2JKJTNJWMTZAP5F4HGA2UEKRHORRYVFKQ2FAX4KLQDVEX7UYQ, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.466681119 (929h59m11.337764154s) [noise=true , address=145.79.227.37:4566 , piece_id=NHE62LRK3ZLPFLQXOIPJZ3LB47RSNU7L2C3DGJKZYV3EXLYBSN6Q, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.468074292 (929h59m11.339157327s) [noise=true , address=145.79.227.11:4732 , piece_id=ZNFPDXJOGVKRQHPN66JHECSZLNEQ57MVWX4GAA6ETQSCNWCJQ75Q, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.472717633 (929h59m11.343800668s) [noise=true , address=145.79.227.35:4483 , piece_id=PH7UUSSVGW3I2KXRBDWEIFHDL3SC63SYDUPWMKG3NX3HKG6E3OYA, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.502968303 (929h59m11.374051338s) [noise=true , address=145.79.227.7:4293 , piece_id=LTQ6DD2KQ55UCWIJ43SWJO7Q5IDDJFTSD2Z7SVMHR2HLUOUSBNDQ, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.554022174 (929h59m11.425105219s) [noise=true , address=145.79.227.51:4409 , piece_id=3FOIHL6K6JAEDPWIRHT7IJOQ534UUKYAVCNAW4V3R5QL3D25UKAA, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.585092324 (929h59m11.456175369s) [noise=true , address=145.79.227.47:4267 , piece_id=RB3RDZAJRF3VV6LL6LUI5544BM7MFTGUIQ64Y4G23FHFRIYWZNUA, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.585114155 (929h59m11.4561972s) [seg_pos={0 0}, upload=813972]: Segment reached optimal threshold of 13 pieces. Jan 13 09:28:11.585293828 (929h59m11.456376863s) [noise=true , address=145.79.227.52:4447 , piece_id=ZZMECYYJKEXTVZI6WQVEMT23OAVZ7SILIHSSE2OHFKSESUAQHJRQ, seg_pos={0 0}, upload=813972]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP): protocol: write tcp 188.166.225.170:44548->145.79.227.52:4447: use of closed network connection; write tcp 188.166.225.170:44548->145.79.227.52:4447: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44548->145.79.227.52:4447: use of closed network connection Jan 13 09:28:11.585299989 (929h59m11.456383025s) [noise=true , address=145.79.227.52:4447 , piece_id=ZZMECYYJKEXTVZI6WQVEMT23OAVZ7SILIHSSE2OHFKSESUAQHJRQ, seg_pos={0 0}, upload=813972]: Long tail cancellation due to optimal threshold Jan 13 09:28:11.585433102 (929h59m11.456516137s) [noise=true , address=145.79.227.6:4257 , piece_id=7NPEYE7VHKUHSOYCDOUVMN2HBL4JEXAGZS64K7M6L6DDIIF755TQ, seg_pos={0 0}, upload=813972]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK): protocol: write tcp 188.166.225.170:52450->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:52450->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52450->145.79.227.6:4257: use of closed network connection Jan 13 09:28:11.585440497 (929h59m11.456523532s) [noise=true , address=145.79.227.6:4257 , piece_id=7NPEYE7VHKUHSOYCDOUVMN2HBL4JEXAGZS64K7M6L6DDIIF755TQ, seg_pos={0 0}, upload=813972]: Long tail cancellation due to optimal threshold Jan 13 09:28:11.585617424 (929h59m11.456700469s) [noise=true , address=145.79.227.49:4337 , piece_id=ZRIX2M4OOA6LKJ6FJ2JORHE4XZQDJRG3KSBUNRJ3EEEXMIK5VMYA, seg_pos={0 0}, upload=813972]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: read tcp 188.166.225.170:49630->145.79.227.49:4337: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 13 09:28:11.585629516 (929h59m11.456712562s) [noise=true , address=145.79.227.49:4337 , piece_id=ZRIX2M4OOA6LKJ6FJ2JORHE4XZQDJRG3KSBUNRJ3EEEXMIK5VMYA, seg_pos={0 0}, upload=813972]: Long tail cancellation due to optimal threshold Jan 13 09:28:11.585663561 (929h59m11.456746596s) [seg_pos={0 0}, upload=813972]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 13 09:28:11.585783911 (929h59m11.456866946s) [upload=813972]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 13 09:28:11.585791646 (929h59m11.456874681s) [upload=813972]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 13 09:28:11.889773771 (929h59m11.760856816s) [upload=813972]: Done uploading segments... Jan 13 09:28:11.889820049 (929h59m11.760903094s) [upload=813972]: Upload finished. err: <nil> Duration: 5.788114042s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......................X....................X.......X..X....] duration: 20.298131177s trace: https://jaeger.dev.storj.io/trace/5b03c94f908e0034 FAIL (2026-01-13 09:54:48 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 20.298147478s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 9.514246652s trace: https://jaeger.dev.storj.io/trace/4226536f67e7860 FAIL (2026-01-13 09:54:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.514342985s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........................X....................X.......X..X...] duration: 24.020157374s trace: https://jaeger.dev.storj.io/trace/5b03c94f908e0034 FAIL (2026-01-13 09:54:27 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 24.020184406s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 5.769687225s trace: https://jaeger.dev.storj.io/trace/209f6f5543c7dce4 FAIL (2026-01-13 09:54:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1vMQm9RufCEcHca9k1JEhyED9MfYjvZcT2MFYic3PmEKyUoC7X, Piece ID: NOBWZTUK4VDKXM3PCO3Y623KIW2XKFRZXWZEDFVVRT5JHTPCZZQQ) context deadline exceeded; piecestore: (Node ID: 12UoCGA4FjE4yADPXL1oifSPJssPgSGh81REZHwGv2eu1Y1kEaj, Piece ID: 3FOIHL6K6JAEDPWIRHT7IJOQ534UUKYAVCNAW4V3R5QL3D25UKAA) context deadline exceeded; piecestore: (Node ID: 1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, Piece ID: KUG2JKJTNJWMTZAP5F4HGA2UEKRHORRYVFKQ2FAX4KLQDVEX7UYQ) context deadline exceeded; piecestore: (Node ID: 12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, Piece ID: LTQ6DD2KQ55UCWIJ43SWJO7Q5IDDJFTSD2Z7SVMHR2HLUOUSBNDQ) context deadline exceeded ======LOGS====== ent. Inline: false Jan 13 09:28:06.315478350 (929h59m6.186561385s) [upload=813972]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jan 13 09:28:06.315483540 (929h59m6.186566575s) [upload=813972]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jan 13 09:28:09.241621500 (929h59m9.112704545s) [seg_pos={0 0}, upload=813972]: Begin upload segment... Jan 13 09:28:09.241671425 (929h59m9.11275446s) [seg_pos={0 0}, upload=813972]: Done begin upload segment. Jan 13 09:28:09.241703947 (929h59m9.112786992s) [noise=true , address=145.79.227.47:4267 , piece_id=RB3RDZAJRF3VV6LL6LUI5544BM7MFTGUIQ64Y4G23FHFRIYWZNUA, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.241712664 (929h59m9.112795699s) [noise=true , address=145.79.227.49:4337 , piece_id=ZRIX2M4OOA6LKJ6FJ2JORHE4XZQDJRG3KSBUNRJ3EEEXMIK5VMYA, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.242438585 (929h59m9.11352162s) [noise=true , address=145.79.227.2:4113 , piece_id=KUG2JKJTNJWMTZAP5F4HGA2UEKRHORRYVFKQ2FAX4KLQDVEX7UYQ, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.242727556 (929h59m9.113810591s) [noise=true , address=145.79.227.35:4483 , piece_id=PH7UUSSVGW3I2KXRBDWEIFHDL3SC63SYDUPWMKG3NX3HKG6E3OYA, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.243047555 (929h59m9.1141306s) [noise=true , address=145.79.227.5:4221 , piece_id=MAYWAFPWGGY6OJMP2OPGXJ4CFWMHIOL32IDBYGFVIONN6PFRDK2Q, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.243340904 (929h59m9.114423939s) [noise=true , address=145.79.227.48:4301 , piece_id=BIBJBBTITB6AA2SWXXZAIEOOAIARHUIVCBXHB4JXMUSELMZSLLTA, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.243735925 (929h59m9.11481897s) [noise=true , address=145.79.227.37:4566 , piece_id=NHE62LRK3ZLPFLQXOIPJZ3LB47RSNU7L2C3DGJKZYV3EXLYBSN6Q, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.244030276 (929h59m9.115113312s) [noise=true , address=145.79.227.11:4732 , piece_id=ZNFPDXJOGVKRQHPN66JHECSZLNEQ57MVWX4GAA6ETQSCNWCJQ75Q, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.245863306 (929h59m9.116946341s) [noise=true , address=145.79.227.7:4293 , piece_id=LTQ6DD2KQ55UCWIJ43SWJO7Q5IDDJFTSD2Z7SVMHR2HLUOUSBNDQ, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.247027153 (929h59m9.118110188s) [noise=true , address=145.79.227.6:4257 , piece_id=7NPEYE7VHKUHSOYCDOUVMN2HBL4JEXAGZS64K7M6L6DDIIF755TQ, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.248440564 (929h59m9.119523619s) [noise=true , address=145.79.227.31:4799 , piece_id=NOBWZTUK4VDKXM3PCO3Y623KIW2XKFRZXWZEDFVVRT5JHTPCZZQQ, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.249844076 (929h59m9.120927121s) [noise=true , address=145.79.227.50:4373 , piece_id=R5A7RCLHAH3ODMP563M27O6T7ZNACL36HJJOFVLRHYG456XUXXBA, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.251275041 (929h59m9.122358076s) [noise=true , address=145.79.227.52:4447 , piece_id=ZZMECYYJKEXTVZI6WQVEMT23OAVZ7SILIHSSE2OHFKSESUAQHJRQ, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.252655870 (929h59m9.123738915s) [noise=true , address=145.79.227.21:4473 , piece_id=FHOGGYMZY7DU6T7FUFNALWHI4Y6SDMPYF242QLW7C4UZIKKXEJ3A, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.253473537 (929h59m9.124556582s) [noise=true , address=145.79.227.13:4803 , piece_id=DNNAZPZ2TYQI6VCXNIZKSMU5VQCXKEDN3L6K6MHSVNRN3R246BCQ, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:09.254758193 (929h59m9.125841238s) [noise=true , address=145.79.227.51:4409 , piece_id=3FOIHL6K6JAEDPWIRHT7IJOQ534UUKYAVCNAW4V3R5QL3D25UKAA, seg_pos={0 0}, upload=813972]: Uploading piece... Jan 13 09:28:11.087868620 (929h59m10.958951665s) [upload=813972]: Next returned nil segment Jan 13 09:28:11.087884400 (929h59m10.958967435s) [upload=813972]: Waiting for error group managing segments... Jan 13 09:28:11.388729734 (929h59m11.259812799s) [noise=true , address=145.79.227.50:4373 , piece_id=R5A7RCLHAH3ODMP563M27O6T7ZNACL36HJJOFVLRHYG456XUXXBA, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.399388468 (929h59m11.270471504s) [noise=true , address=145.79.227.21:4473 , piece_id=FHOGGYMZY7DU6T7FUFNALWHI4Y6SDMPYF242QLW7C4UZIKKXEJ3A, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.405180828 (929h59m11.276263873s) [noise=true , address=145.79.227.13:4803 , piece_id=DNNAZPZ2TYQI6VCXNIZKSMU5VQCXKEDN3L6K6MHSVNRN3R246BCQ, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.423915039 (929h59m11.294998074s) [noise=true , address=145.79.227.31:4799 , piece_id=NOBWZTUK4VDKXM3PCO3Y623KIW2XKFRZXWZEDFVVRT5JHTPCZZQQ, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.425205065 (929h59m11.29628811s) [noise=true , address=145.79.227.48:4301 , piece_id=BIBJBBTITB6AA2SWXXZAIEOOAIARHUIVCBXHB4JXMUSELMZSLLTA, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.427897582 (929h59m11.298980617s) [noise=true , address=145.79.227.5:4221 , piece_id=MAYWAFPWGGY6OJMP2OPGXJ4CFWMHIOL32IDBYGFVIONN6PFRDK2Q, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.461421214 (929h59m11.332504249s) [noise=true , address=145.79.227.2:4113 , piece_id=KUG2JKJTNJWMTZAP5F4HGA2UEKRHORRYVFKQ2FAX4KLQDVEX7UYQ, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.466681119 (929h59m11.337764154s) [noise=true , address=145.79.227.37:4566 , piece_id=NHE62LRK3ZLPFLQXOIPJZ3LB47RSNU7L2C3DGJKZYV3EXLYBSN6Q, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.468074292 (929h59m11.339157327s) [noise=true , address=145.79.227.11:4732 , piece_id=ZNFPDXJOGVKRQHPN66JHECSZLNEQ57MVWX4GAA6ETQSCNWCJQ75Q, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.472717633 (929h59m11.343800668s) [noise=true , address=145.79.227.35:4483 , piece_id=PH7UUSSVGW3I2KXRBDWEIFHDL3SC63SYDUPWMKG3NX3HKG6E3OYA, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.502968303 (929h59m11.374051338s) [noise=true , address=145.79.227.7:4293 , piece_id=LTQ6DD2KQ55UCWIJ43SWJO7Q5IDDJFTSD2Z7SVMHR2HLUOUSBNDQ, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.554022174 (929h59m11.425105219s) [noise=true , address=145.79.227.51:4409 , piece_id=3FOIHL6K6JAEDPWIRHT7IJOQ534UUKYAVCNAW4V3R5QL3D25UKAA, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.585092324 (929h59m11.456175369s) [noise=true , address=145.79.227.47:4267 , piece_id=RB3RDZAJRF3VV6LL6LUI5544BM7MFTGUIQ64Y4G23FHFRIYWZNUA, seg_pos={0 0}, upload=813972]: Done uploading piece. err: <nil> Jan 13 09:28:11.585114155 (929h59m11.4561972s) [seg_pos={0 0}, upload=813972]: Segment reached optimal threshold of 13 pieces. Jan 13 09:28:11.585293828 (929h59m11.456376863s) [noise=true , address=145.79.227.52:4447 , piece_id=ZZMECYYJKEXTVZI6WQVEMT23OAVZ7SILIHSSE2OHFKSESUAQHJRQ, seg_pos={0 0}, upload=813972]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fqT9xA7kXTG8K9XwTUxL6FXibV35AkPHDK6grKGHg17fk91fP): protocol: write tcp 188.166.225.170:44548->145.79.227.52:4447: use of closed network connection; write tcp 188.166.225.170:44548->145.79.227.52:4447: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44548->145.79.227.52:4447: use of closed network connection Jan 13 09:28:11.585299989 (929h59m11.456383025s) [noise=true , address=145.79.227.52:4447 , piece_id=ZZMECYYJKEXTVZI6WQVEMT23OAVZ7SILIHSSE2OHFKSESUAQHJRQ, seg_pos={0 0}, upload=813972]: Long tail cancellation due to optimal threshold Jan 13 09:28:11.585433102 (929h59m11.456516137s) [noise=true , address=145.79.227.6:4257 , piece_id=7NPEYE7VHKUHSOYCDOUVMN2HBL4JEXAGZS64K7M6L6DDIIF755TQ, seg_pos={0 0}, upload=813972]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK): protocol: write tcp 188.166.225.170:52450->145.79.227.6:4257: use of closed network connection; write tcp 188.166.225.170:52450->145.79.227.6:4257: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52450->145.79.227.6:4257: use of closed network connection Jan 13 09:28:11.585440497 (929h59m11.456523532s) [noise=true , address=145.79.227.6:4257 , piece_id=7NPEYE7VHKUHSOYCDOUVMN2HBL4JEXAGZS64K7M6L6DDIIF755TQ, seg_pos={0 0}, upload=813972]: Long tail cancellation due to optimal threshold Jan 13 09:28:11.585617424 (929h59m11.456700469s) [noise=true , address=145.79.227.49:4337 , piece_id=ZRIX2M4OOA6LKJ6FJ2JORHE4XZQDJRG3KSBUNRJ3EEEXMIK5VMYA, seg_pos={0 0}, upload=813972]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1nTZFfzNCcVAjut7NXf9seifysJDfvXkS8aZSnXKfw6gh1TrjE): protocol: read tcp 188.166.225.170:49630->145.79.227.49:4337: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jan 13 09:28:11.585629516 (929h59m11.456712562s) [noise=true , address=145.79.227.49:4337 , piece_id=ZRIX2M4OOA6LKJ6FJ2JORHE4XZQDJRG3KSBUNRJ3EEEXMIK5VMYA, seg_pos={0 0}, upload=813972]: Long tail cancellation due to optimal threshold Jan 13 09:28:11.585663561 (929h59m11.456746596s) [seg_pos={0 0}, upload=813972]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jan 13 09:28:11.585783911 (929h59m11.456866946s) [upload=813972]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jan 13 09:28:11.585791646 (929h59m11.456874681s) [upload=813972]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jan 13 09:28:11.889773771 (929h59m11.760856816s) [upload=813972]: Done uploading segments... Jan 13 09:28:11.889820049 (929h59m11.760903094s) [upload=813972]: Upload finished. err: <nil> Duration: 5.769727762s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 58 out of 60 is failed (threshold=6) [XXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXX] duration: 5.001529449s trace: https://jaeger.dev.storj.io/trace/387da93632caf813 FAIL (2026-01-13 09:54:03 UTC) orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001552903s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)