Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [.X........................................X........X........] duration:
5.936947937s trace:
https://jaeger.dev.storj.io/trace/1ea01aa868761fdc
FAIL (2026-01-07 07:17:07 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
133194519 (783h41m50.004277565s) [noise=true , address=145.79.227.32:4851 , piece_id=5W74ZJOGSKONFPVTJJPHDWNFHLTWRU4KEVX25X4PQCKTRJZR3CZA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.133698059 (783h41m50.004781104s) [noise=true , address=145.79.227.33:4897 , piece_id=NLJJNXY2DGHMZWGJRSKGTYDVYZLYZJSVYVIZ35IUCMSI34RD5TBA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.134246032 (783h41m50.005329067s) [noise=true , address=145.79.227.44:4227 , piece_id=TCERBF6EX3JHAV63JYLUYA54OHYY33YEM25EQ6CIRS4746KNCULA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.134617359 (783h41m50.005700404s) [noise=true , address=145.79.227.30:4795 , piece_id=G7EGUQIODKUJDE6FEWO5B5C3RQYQ37SAHQYC6BI7X4JS6A7VMOXQ, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.135018602 (783h41m50.006101637s) [noise=true , address=77.90.31.16:4741 , piece_id=G4TPC522KLMJBYHYW2SYTUMJFVP4K54R63ZIHOCPQINQO5WE7Q2Q, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.135492385 (783h41m50.00657542s) [noise=true , address=145.79.227.41:4092 , piece_id=3WQEAVSBFU5KF6WVNGACEVQ6M42JEMWW6OHCXPCXVNZCOG4XSMGQ, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.135895753 (783h41m50.006978799s) [noise=true , address=145.79.227.34:4926 , piece_id=AUBI2PYIJK5YTKFZHL5WF7BQOKNKEPSTZQBERRGEOFZN7RE7T2SA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.136238677 (783h41m50.007321722s) [noise=true , address=193.149.164.143:4699 , piece_id=DFHC3CPJBT2U7MW5HSNGIGTPMD3QZD4ON62UJ2HNNVWRXOJ4A6PQ, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.136546201 (783h41m50.007629236s) [noise=true , address=145.79.227.36:4543 , piece_id=MUK6THUJD4QHK4VLL3R6DSZP47VV5RJAUPKG6RTPR7F6GTIC6GIA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.136565809 (783h41m50.007648844s) [noise=true , address=145.79.227.29:4946 , piece_id=FR3BLE7ANKJYAQG5GZJ4RL4543FQ55K7VHSJMQGCLC7UDRNKCASA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.137150974 (783h41m50.008234019s) [noise=true , address=145.79.227.35:4487 , piece_id=E5X6GYKRPSYHASZXZB62XA3LXB6EJPTMENDRW6BGEDVDUD5BW4TA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.137656586 (783h41m50.008739621s) [noise=true , address=77.90.31.11:4584 , piece_id=LIDDXUURY5ARSGQ6ETUFION36NJFWMIGJHOSOIE4ZUXJPTJNCGSA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.138273391 (783h41m50.009356436s) [noise=true , address=64.49.13.169:4713 , piece_id=6JKGRVL7KQOZVIIJM3OFQUHWUEF42KMW7Y6I5EPVKVRTKEQVBFNQ, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.138867362 (783h41m50.009950407s) [noise=true , address=64.49.13.155:4916 , piece_id=FPLKLM4RLSO54V7XX3YZF7CGXWSGKNBAKEYQ5FYFYHVE7CTS6PIA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.139440193 (783h41m50.010523238s) [noise=true , address=145.79.227.49:4372 , piece_id=HQ535NWA7OLGPNMYWPMHDJL3ICULP4GEXMYTKLDTC2SPMQYCRZOQ, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.139988797 (783h41m50.011071832s) [noise=true , address=45.91.116.10:4110 , piece_id=VODTQS7BAXPWLTDDTTW75FXYRDBGM2PSJQTIGCRCRNAMMTE55LJA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.140847663 (783h41m50.011930698s) [noise=true , address=45.91.116.14:4200 , piece_id=ON5HNVYRNJTJCNV4JU6NI2ITS2VKZNNMYOBZUZQ7OLXPYAUQL2JQ, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:51.568606354 (783h41m51.439689389s) [upload=685546]: Next returned nil segment
Jan 7 07:10:51.568621312 (783h41m51.439704347s) [upload=685546]: Waiting for error group managing segments...
Jan 7 07:10:51.990916323 (783h41m51.861999368s) [noise=true , address=64.49.13.155:4916 , piece_id=FPLKLM4RLSO54V7XX3YZF7CGXWSGKNBAKEYQ5FYFYHVE7CTS6PIA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:51.994944574 (783h41m51.866027619s) [noise=true , address=145.79.227.44:4227 , piece_id=TCERBF6EX3JHAV63JYLUYA54OHYY33YEM25EQ6CIRS4746KNCULA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.000551040 (783h41m51.871634105s) [noise=true , address=145.79.227.41:4092 , piece_id=3WQEAVSBFU5KF6WVNGACEVQ6M42JEMWW6OHCXPCXVNZCOG4XSMGQ, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.005752454 (783h41m51.876835509s) [noise=true , address=145.79.227.30:4795 , piece_id=G7EGUQIODKUJDE6FEWO5B5C3RQYQ37SAHQYC6BI7X4JS6A7VMOXQ, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.006180148 (783h41m51.877263203s) [noise=true , address=145.79.227.32:4851 , piece_id=5W74ZJOGSKONFPVTJJPHDWNFHLTWRU4KEVX25X4PQCKTRJZR3CZA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.062554132 (783h41m51.933637177s) [noise=true , address=64.49.13.169:4713 , piece_id=6JKGRVL7KQOZVIIJM3OFQUHWUEF42KMW7Y6I5EPVKVRTKEQVBFNQ, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.070238635 (783h41m51.94132167s) [noise=true , address=145.79.227.34:4926 , piece_id=AUBI2PYIJK5YTKFZHL5WF7BQOKNKEPSTZQBERRGEOFZN7RE7T2SA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.088972997 (783h41m51.960056032s) [noise=true , address=145.79.227.33:4897 , piece_id=NLJJNXY2DGHMZWGJRSKGTYDVYZLYZJSVYVIZ35IUCMSI34RD5TBA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.281595960 (783h41m52.152678995s) [noise=true , address=77.90.31.12:4603 , piece_id=HPBI4VUKMT2KFKRW2JTX32HB2YGDGLB2V4BFRVVQ2725Y2IC2LOA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.356241606 (783h41m52.227324641s) [noise=true , address=77.90.31.11:4584 , piece_id=LIDDXUURY5ARSGQ6ETUFION36NJFWMIGJHOSOIE4ZUXJPTJNCGSA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.679275149 (783h41m52.550358204s) [noise=true , address=45.91.116.14:4200 , piece_id=ON5HNVYRNJTJCNV4JU6NI2ITS2VKZNNMYOBZUZQ7OLXPYAUQL2JQ, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.848093704 (783h41m52.719176749s) [noise=true , address=193.149.164.143:4699 , piece_id=DFHC3CPJBT2U7MW5HSNGIGTPMD3QZD4ON62UJ2HNNVWRXOJ4A6PQ, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:53.022732271 (783h41m52.893815316s) [noise=true , address=145.79.227.49:4372 , piece_id=HQ535NWA7OLGPNMYWPMHDJL3ICULP4GEXMYTKLDTC2SPMQYCRZOQ, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:53.048522630 (783h41m52.919605675s) [noise=true , address=145.79.227.36:4543 , piece_id=MUK6THUJD4QHK4VLL3R6DSZP47VV5RJAUPKG6RTPR7F6GTIC6GIA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:53.048547036 (783h41m52.919630082s) [seg_pos={0 0}, upload=685546]: Segment reached optimal threshold of 14 pieces.
Jan 7 07:10:53.048761935 (783h41m52.919844971s) [noise=true , address=77.90.31.16:4741 , piece_id=G4TPC522KLMJBYHYW2SYTUMJFVP4K54R63ZIHOCPQINQO5WE7Q2Q, seg_pos={0 0}, upload=685546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR): protocol: expected piece hash; context canceled; EOF
Jan 7 07:10:53.048771714 (783h41m52.919854749s) [noise=true , address=77.90.31.16:4741 , piece_id=G4TPC522KLMJBYHYW2SYTUMJFVP4K54R63ZIHOCPQINQO5WE7Q2Q, seg_pos={0 0}, upload=685546]: Long tail cancellation due to optimal threshold
Jan 7 07:10:53.049042570 (783h41m52.920125605s) [noise=true , address=145.79.227.29:4946 , piece_id=FR3BLE7ANKJYAQG5GZJ4RL4543FQ55K7VHSJMQGCLC7UDRNKCASA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12v9Q7aigjWPNXyshHmQt9YKrDu5af1vd1TxkDeHeKixSrWgrLX): protocol: expected piece hash; context canceled; EOF
Jan 7 07:10:53.049049483 (783h41m52.920132518s) [noise=true , address=145.79.227.29:4946 , piece_id=FR3BLE7ANKJYAQG5GZJ4RL4543FQ55K7VHSJMQGCLC7UDRNKCASA, seg_pos={0 0}, upload=685546]: Long tail cancellation due to optimal threshold
Jan 7 07:10:53.049146277 (783h41m52.920229322s) [noise=true , address=145.79.227.35:4487 , piece_id=E5X6GYKRPSYHASZXZB62XA3LXB6EJPTMENDRW6BGEDVDUD5BW4TA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AdZLDV7UBbV2DimCHzmoNxycLigvLMx6Fy9U9QB35WTtiV1KX): protocol: expected piece hash; context canceled; EOF
Jan 7 07:10:53.049151166 (783h41m52.920234212s) [noise=true , address=145.79.227.35:4487 , piece_id=E5X6GYKRPSYHASZXZB62XA3LXB6EJPTMENDRW6BGEDVDUD5BW4TA, seg_pos={0 0}, upload=685546]: Long tail cancellation due to optimal threshold
Jan 7 07:10:53.049299398 (783h41m52.920382434s) [noise=true , address=45.91.116.10:4110 , piece_id=VODTQS7BAXPWLTDDTTW75FXYRDBGM2PSJQTIGCRCRNAMMTE55LJA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123KwusATu1ZKtNNVHZZZBE67ZEnVb8puyfQidBi3fEWREqdf1h): protocol: expected piece hash; context canceled; EOF
Jan 7 07:10:53.049308466 (783h41m52.920391512s) [noise=true , address=45.91.116.10:4110 , piece_id=VODTQS7BAXPWLTDDTTW75FXYRDBGM2PSJQTIGCRCRNAMMTE55LJA, seg_pos={0 0}, upload=685546]: Long tail cancellation due to optimal threshold
Jan 7 07:10:53.049324507 (783h41m52.920407542s) [seg_pos={0 0}, upload=685546]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 7 07:10:53.049388278 (783h41m52.920471313s) [upload=685546]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 7 07:10:53.049392767 (783h41m52.920475812s) [upload=685546]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 7 07:10:53.340338951 (783h41m53.211421996s) [upload=685546]: Done uploading segments...
Jan 7 07:10:53.340359300 (783h41m53.211442335s) [upload=685546]: Upload finished. err: <nil>
Duration: 5.936978697s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [.....X.XXX.X.X..X....X.X.....X....X.X.....X.......X....XX...] duration:
25.17345367s trace:
https://jaeger.dev.storj.io/trace/3649792573630e65
FAIL (2026-01-07 07:17:04 UTC)
orbiter warning: context deadline exceeded
Duration: 25.173477265s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [............X...............X...................X...........] duration:
34.582410729s trace:
https://jaeger.dev.storj.io/trace/36161e151ba11c34
FAIL (2026-01-07 07:17:03 UTC)
orbiter warning: context deadline exceeded
Duration: 34.582433342s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
15 out of 60 is failed (threshold=3) [X.X...X....X.........X..X.X..X.......XX.X....X....XX....X...] duration:
5.703564543s trace:
https://jaeger.dev.storj.io/trace/566429788081faee
FAIL (2026-01-07 07:16:55 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
) [upload=685582]: Uploading segments...
Jan 7 07:13:32.990108051 (783h44m32.861191096s) [upload=685582]: Got next segment. Inline: false
Jan 7 07:13:32.990479758 (783h44m32.861562803s) [upload=685582]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 7 07:13:32.990486191 (783h44m32.861569226s) [upload=685582]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 7 07:13:33.230730672 (783h44m33.101813717s) [seg_pos={0 0}, upload=685582]: Begin upload segment...
Jan 7 07:13:33.230783873 (783h44m33.101866908s) [seg_pos={0 0}, upload=685582]: Done begin upload segment.
Jan 7 07:13:33.230814070 (783h44m33.101897115s) [noise=true , address=45.91.116.13:4153 , piece_id=27PHIESDJJJKDKJITJODFYQLSF42BX3BVTMOXFJAUGK73D3Z47UA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.230992941 (783h44m33.102075976s) [noise=true , address=45.91.116.5:4300 , piece_id=5FMCKFKIRBUKZGZTTBKCMURBHZTBKQPHEZMHJWJHLM2PLEDFNXSA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.231512250 (783h44m33.102595285s) [noise=true , address=145.79.227.22:4511 , piece_id=VHGXFDPNKXDZI2ARWI7227A5CF3P572VJSQZS75BMMHPAVCPPO7Q, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.231639272 (783h44m33.102722307s) [noise=true , address=145.79.227.4:4185 , piece_id=SMLAUFNVX6M24OXQ2PK2TIP5MXF3KZYRGHBFXYN7OXD5L76PCY3Q, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.231935536 (783h44m33.103018571s) [noise=true , address=145.79.227.31:4798 , piece_id=TKL7NRLI7DYTXU4HEOBJHXQR5CGHU7IG53L2GLENDCORGVVN2KOQ, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.232203125 (783h44m33.10328616s) [noise=true , address=145.79.227.38:4978 , piece_id=QV3BBDP5GVUMTBI2HIE7FWUIUOQVKIOO2D6OJNW2VDCXDFKMISUA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.232580194 (783h44m33.103663229s) [noise=true , address=145.79.227.52:4445 , piece_id=D6QKUQIBGUGV3V2ZELIM6U5UMTHT2765CHQBIKQ2N3BJF3UXH3XA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.232942814 (783h44m33.104025849s) [noise=true , address=45.91.116.8:4047 , piece_id=D6ZXOD5IBAYIVGUWNC6OGIL3OTB7JF6OAFGNGIR2RH7NDWRAEORA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.233250680 (783h44m33.104333725s) [noise=true , address=145.79.227.40:4049 , piece_id=RV43PFYU6UWU5QFHCO7T6F3E3VBNEYQRMA4UCPXT54PP3T3ZD62Q, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.233527197 (783h44m33.104610242s) [noise=true , address=77.90.31.16:4736 , piece_id=6I7NM23KI67OH2RKW4NSGYGJYWZXMH5HVLHLOQFVNCSFW4WZJ5NA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.234521901 (783h44m33.105604936s) [noise=true , address=45.91.116.12:4226 , piece_id=V7NKTZUGNA2HMXA3FSYMIXMSDLQVEUTJB74IW32GYF4XTU45UY6Q, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.235153884 (783h44m33.106236919s) [noise=true , address=145.79.227.45:4231 , piece_id=ESQLZIZB2AVESVJRHVGX4T5AJDEKVGJEJ3JSGA5J25BOQ4UUF7KA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.235615984 (783h44m33.106699019s) [noise=true , address=145.79.227.11:4729 , piece_id=5EQJ3NL4R45GFF35F2WOYWCD77UGR6XUV6B4TIZASZBCBCX6APWA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.236105967 (783h44m33.107189002s) [noise=true , address=145.79.227.5:4221 , piece_id=SHQ2ARJ6XTS7NHVSSNAXW2VR7BPS26MSX76INSRBH6KXPM4SQKEA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.236621799 (783h44m33.107704844s) [noise=true , address=145.79.227.7:4293 , piece_id=WPVYNVVDT4G5DG2HIOLUDVJNIU5O3WFII2H3GIHEZDBST4F2DKGA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.239024364 (783h44m33.110107409s) [noise=true , address=77.90.31.18:4805 , piece_id=YIHVN23OWQTVKXHZIAY4K2C65LU67NA7C6GCC63PXZHCWAZGL7DQ, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:34.837564819 (783h44m34.708647854s) [upload=685582]: Next returned nil segment
Jan 7 07:13:34.837594546 (783h44m34.708677581s) [upload=685582]: Waiting for error group managing segments...
Jan 7 07:13:35.221987466 (783h44m35.093070501s) [noise=true , address=145.79.227.38:4978 , piece_id=QV3BBDP5GVUMTBI2HIE7FWUIUOQVKIOO2D6OJNW2VDCXDFKMISUA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.224032561 (783h44m35.095115596s) [noise=true , address=145.79.227.11:4729 , piece_id=5EQJ3NL4R45GFF35F2WOYWCD77UGR6XUV6B4TIZASZBCBCX6APWA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.224625309 (783h44m35.095708355s) [noise=true , address=145.79.227.52:4445 , piece_id=D6QKUQIBGUGV3V2ZELIM6U5UMTHT2765CHQBIKQ2N3BJF3UXH3XA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.225249578 (783h44m35.096332613s) [noise=true , address=145.79.227.22:4511 , piece_id=VHGXFDPNKXDZI2ARWI7227A5CF3P572VJSQZS75BMMHPAVCPPO7Q, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.233155391 (783h44m35.104238446s) [noise=true , address=145.79.227.45:4231 , piece_id=ESQLZIZB2AVESVJRHVGX4T5AJDEKVGJEJ3JSGA5J25BOQ4UUF7KA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.244418267 (783h44m35.115501302s) [noise=true , address=145.79.227.5:4221 , piece_id=SHQ2ARJ6XTS7NHVSSNAXW2VR7BPS26MSX76INSRBH6KXPM4SQKEA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.253004497 (783h44m35.124087542s) [noise=true , address=145.79.227.31:4798 , piece_id=TKL7NRLI7DYTXU4HEOBJHXQR5CGHU7IG53L2GLENDCORGVVN2KOQ, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.260994110 (783h44m35.132077145s) [noise=true , address=145.79.227.7:4293 , piece_id=WPVYNVVDT4G5DG2HIOLUDVJNIU5O3WFII2H3GIHEZDBST4F2DKGA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.298671554 (783h44m35.169754589s) [noise=true , address=145.79.227.4:4185 , piece_id=SMLAUFNVX6M24OXQ2PK2TIP5MXF3KZYRGHBFXYN7OXD5L76PCY3Q, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.306366076 (783h44m35.177449111s) [noise=true , address=145.79.227.40:4049 , piece_id=RV43PFYU6UWU5QFHCO7T6F3E3VBNEYQRMA4UCPXT54PP3T3ZD62Q, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.519691228 (783h44m35.390774263s) [noise=true , address=77.90.31.16:4736 , piece_id=6I7NM23KI67OH2RKW4NSGYGJYWZXMH5HVLHLOQFVNCSFW4WZJ5NA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.749540043 (783h44m35.620623078s) [noise=true , address=45.91.116.5:4300 , piece_id=5FMCKFKIRBUKZGZTTBKCMURBHZTBKQPHEZMHJWJHLM2PLEDFNXSA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.823582121 (783h44m35.694665156s) [noise=true , address=45.91.116.8:4047 , piece_id=D6ZXOD5IBAYIVGUWNC6OGIL3OTB7JF6OAFGNGIR2RH7NDWRAEORA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.823600715 (783h44m35.69468375s) [seg_pos={0 0}, upload=685582]: Segment reached optimal threshold of 13 pieces.
Jan 7 07:13:35.824004615 (783h44m35.69508765s) [noise=true , address=45.91.116.12:4226 , piece_id=V7NKTZUGNA2HMXA3FSYMIXMSDLQVEUTJB74IW32GYF4XTU45UY6Q, seg_pos={0 0}, upload=685582]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: write tcp 188.166.225.170:54196->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:54196->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54196->45.91.116.12:4226: use of closed network connection
Jan 7 07:13:35.824012139 (783h44m35.695095174s) [noise=true , address=45.91.116.12:4226 , piece_id=V7NKTZUGNA2HMXA3FSYMIXMSDLQVEUTJB74IW32GYF4XTU45UY6Q, seg_pos={0 0}, upload=685582]: Long tail cancellation due to optimal threshold
Jan 7 07:13:35.824197492 (783h44m35.695280527s) [noise=true , address=45.91.116.13:4153 , piece_id=27PHIESDJJJKDKJITJODFYQLSF42BX3BVTMOXFJAUGK73D3Z47UA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: read tcp 188.166.225.170:33886->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 7 07:13:35.824205007 (783h44m35.695288042s) [noise=true , address=45.91.116.13:4153 , piece_id=27PHIESDJJJKDKJITJODFYQLSF42BX3BVTMOXFJAUGK73D3Z47UA, seg_pos={0 0}, upload=685582]: Long tail cancellation due to optimal threshold
Jan 7 07:13:35.824576704 (783h44m35.695659739s) [noise=true , address=77.90.31.18:4805 , piece_id=YIHVN23OWQTVKXHZIAY4K2C65LU67NA7C6GCC63PXZHCWAZGL7DQ, seg_pos={0 0}, upload=685582]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: read tcp 188.166.225.170:38598->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 7 07:13:35.824596592 (783h44m35.695679637s) [noise=true , address=77.90.31.18:4805 , piece_id=YIHVN23OWQTVKXHZIAY4K2C65LU67NA7C6GCC63PXZHCWAZGL7DQ, seg_pos={0 0}, upload=685582]: Long tail cancellation due to optimal threshold
Jan 7 07:13:35.824616139 (783h44m35.695699184s) [seg_pos={0 0}, upload=685582]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 7 07:13:35.824682215 (783h44m35.69576526s) [upload=685582]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 7 07:13:35.824686843 (783h44m35.695769878s) [upload=685582]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 7 07:13:36.105700746 (783h44m35.976783791s) [upload=685582]: Done uploading segments...
Jan 7 07:13:36.105723649 (783h44m35.976806684s) [upload=685582]: Upload finished. err: <nil>
Duration: 5.703588801s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [..X........................................X........X.......] duration:
38.145319332s trace:
https://jaeger.dev.storj.io/trace/1ea01aa868761fdc
FAIL (2026-01-07 07:16:53 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
133194519 (783h41m50.004277565s) [noise=true , address=145.79.227.32:4851 , piece_id=5W74ZJOGSKONFPVTJJPHDWNFHLTWRU4KEVX25X4PQCKTRJZR3CZA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.133698059 (783h41m50.004781104s) [noise=true , address=145.79.227.33:4897 , piece_id=NLJJNXY2DGHMZWGJRSKGTYDVYZLYZJSVYVIZ35IUCMSI34RD5TBA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.134246032 (783h41m50.005329067s) [noise=true , address=145.79.227.44:4227 , piece_id=TCERBF6EX3JHAV63JYLUYA54OHYY33YEM25EQ6CIRS4746KNCULA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.134617359 (783h41m50.005700404s) [noise=true , address=145.79.227.30:4795 , piece_id=G7EGUQIODKUJDE6FEWO5B5C3RQYQ37SAHQYC6BI7X4JS6A7VMOXQ, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.135018602 (783h41m50.006101637s) [noise=true , address=77.90.31.16:4741 , piece_id=G4TPC522KLMJBYHYW2SYTUMJFVP4K54R63ZIHOCPQINQO5WE7Q2Q, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.135492385 (783h41m50.00657542s) [noise=true , address=145.79.227.41:4092 , piece_id=3WQEAVSBFU5KF6WVNGACEVQ6M42JEMWW6OHCXPCXVNZCOG4XSMGQ, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.135895753 (783h41m50.006978799s) [noise=true , address=145.79.227.34:4926 , piece_id=AUBI2PYIJK5YTKFZHL5WF7BQOKNKEPSTZQBERRGEOFZN7RE7T2SA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.136238677 (783h41m50.007321722s) [noise=true , address=193.149.164.143:4699 , piece_id=DFHC3CPJBT2U7MW5HSNGIGTPMD3QZD4ON62UJ2HNNVWRXOJ4A6PQ, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.136546201 (783h41m50.007629236s) [noise=true , address=145.79.227.36:4543 , piece_id=MUK6THUJD4QHK4VLL3R6DSZP47VV5RJAUPKG6RTPR7F6GTIC6GIA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.136565809 (783h41m50.007648844s) [noise=true , address=145.79.227.29:4946 , piece_id=FR3BLE7ANKJYAQG5GZJ4RL4543FQ55K7VHSJMQGCLC7UDRNKCASA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.137150974 (783h41m50.008234019s) [noise=true , address=145.79.227.35:4487 , piece_id=E5X6GYKRPSYHASZXZB62XA3LXB6EJPTMENDRW6BGEDVDUD5BW4TA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.137656586 (783h41m50.008739621s) [noise=true , address=77.90.31.11:4584 , piece_id=LIDDXUURY5ARSGQ6ETUFION36NJFWMIGJHOSOIE4ZUXJPTJNCGSA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.138273391 (783h41m50.009356436s) [noise=true , address=64.49.13.169:4713 , piece_id=6JKGRVL7KQOZVIIJM3OFQUHWUEF42KMW7Y6I5EPVKVRTKEQVBFNQ, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.138867362 (783h41m50.009950407s) [noise=true , address=64.49.13.155:4916 , piece_id=FPLKLM4RLSO54V7XX3YZF7CGXWSGKNBAKEYQ5FYFYHVE7CTS6PIA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.139440193 (783h41m50.010523238s) [noise=true , address=145.79.227.49:4372 , piece_id=HQ535NWA7OLGPNMYWPMHDJL3ICULP4GEXMYTKLDTC2SPMQYCRZOQ, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.139988797 (783h41m50.011071832s) [noise=true , address=45.91.116.10:4110 , piece_id=VODTQS7BAXPWLTDDTTW75FXYRDBGM2PSJQTIGCRCRNAMMTE55LJA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.140847663 (783h41m50.011930698s) [noise=true , address=45.91.116.14:4200 , piece_id=ON5HNVYRNJTJCNV4JU6NI2ITS2VKZNNMYOBZUZQ7OLXPYAUQL2JQ, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:51.568606354 (783h41m51.439689389s) [upload=685546]: Next returned nil segment
Jan 7 07:10:51.568621312 (783h41m51.439704347s) [upload=685546]: Waiting for error group managing segments...
Jan 7 07:10:51.990916323 (783h41m51.861999368s) [noise=true , address=64.49.13.155:4916 , piece_id=FPLKLM4RLSO54V7XX3YZF7CGXWSGKNBAKEYQ5FYFYHVE7CTS6PIA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:51.994944574 (783h41m51.866027619s) [noise=true , address=145.79.227.44:4227 , piece_id=TCERBF6EX3JHAV63JYLUYA54OHYY33YEM25EQ6CIRS4746KNCULA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.000551040 (783h41m51.871634105s) [noise=true , address=145.79.227.41:4092 , piece_id=3WQEAVSBFU5KF6WVNGACEVQ6M42JEMWW6OHCXPCXVNZCOG4XSMGQ, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.005752454 (783h41m51.876835509s) [noise=true , address=145.79.227.30:4795 , piece_id=G7EGUQIODKUJDE6FEWO5B5C3RQYQ37SAHQYC6BI7X4JS6A7VMOXQ, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.006180148 (783h41m51.877263203s) [noise=true , address=145.79.227.32:4851 , piece_id=5W74ZJOGSKONFPVTJJPHDWNFHLTWRU4KEVX25X4PQCKTRJZR3CZA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.062554132 (783h41m51.933637177s) [noise=true , address=64.49.13.169:4713 , piece_id=6JKGRVL7KQOZVIIJM3OFQUHWUEF42KMW7Y6I5EPVKVRTKEQVBFNQ, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.070238635 (783h41m51.94132167s) [noise=true , address=145.79.227.34:4926 , piece_id=AUBI2PYIJK5YTKFZHL5WF7BQOKNKEPSTZQBERRGEOFZN7RE7T2SA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.088972997 (783h41m51.960056032s) [noise=true , address=145.79.227.33:4897 , piece_id=NLJJNXY2DGHMZWGJRSKGTYDVYZLYZJSVYVIZ35IUCMSI34RD5TBA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.281595960 (783h41m52.152678995s) [noise=true , address=77.90.31.12:4603 , piece_id=HPBI4VUKMT2KFKRW2JTX32HB2YGDGLB2V4BFRVVQ2725Y2IC2LOA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.356241606 (783h41m52.227324641s) [noise=true , address=77.90.31.11:4584 , piece_id=LIDDXUURY5ARSGQ6ETUFION36NJFWMIGJHOSOIE4ZUXJPTJNCGSA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.679275149 (783h41m52.550358204s) [noise=true , address=45.91.116.14:4200 , piece_id=ON5HNVYRNJTJCNV4JU6NI2ITS2VKZNNMYOBZUZQ7OLXPYAUQL2JQ, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.848093704 (783h41m52.719176749s) [noise=true , address=193.149.164.143:4699 , piece_id=DFHC3CPJBT2U7MW5HSNGIGTPMD3QZD4ON62UJ2HNNVWRXOJ4A6PQ, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:53.022732271 (783h41m52.893815316s) [noise=true , address=145.79.227.49:4372 , piece_id=HQ535NWA7OLGPNMYWPMHDJL3ICULP4GEXMYTKLDTC2SPMQYCRZOQ, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:53.048522630 (783h41m52.919605675s) [noise=true , address=145.79.227.36:4543 , piece_id=MUK6THUJD4QHK4VLL3R6DSZP47VV5RJAUPKG6RTPR7F6GTIC6GIA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:53.048547036 (783h41m52.919630082s) [seg_pos={0 0}, upload=685546]: Segment reached optimal threshold of 14 pieces.
Jan 7 07:10:53.048761935 (783h41m52.919844971s) [noise=true , address=77.90.31.16:4741 , piece_id=G4TPC522KLMJBYHYW2SYTUMJFVP4K54R63ZIHOCPQINQO5WE7Q2Q, seg_pos={0 0}, upload=685546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR): protocol: expected piece hash; context canceled; EOF
Jan 7 07:10:53.048771714 (783h41m52.919854749s) [noise=true , address=77.90.31.16:4741 , piece_id=G4TPC522KLMJBYHYW2SYTUMJFVP4K54R63ZIHOCPQINQO5WE7Q2Q, seg_pos={0 0}, upload=685546]: Long tail cancellation due to optimal threshold
Jan 7 07:10:53.049042570 (783h41m52.920125605s) [noise=true , address=145.79.227.29:4946 , piece_id=FR3BLE7ANKJYAQG5GZJ4RL4543FQ55K7VHSJMQGCLC7UDRNKCASA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12v9Q7aigjWPNXyshHmQt9YKrDu5af1vd1TxkDeHeKixSrWgrLX): protocol: expected piece hash; context canceled; EOF
Jan 7 07:10:53.049049483 (783h41m52.920132518s) [noise=true , address=145.79.227.29:4946 , piece_id=FR3BLE7ANKJYAQG5GZJ4RL4543FQ55K7VHSJMQGCLC7UDRNKCASA, seg_pos={0 0}, upload=685546]: Long tail cancellation due to optimal threshold
Jan 7 07:10:53.049146277 (783h41m52.920229322s) [noise=true , address=145.79.227.35:4487 , piece_id=E5X6GYKRPSYHASZXZB62XA3LXB6EJPTMENDRW6BGEDVDUD5BW4TA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AdZLDV7UBbV2DimCHzmoNxycLigvLMx6Fy9U9QB35WTtiV1KX): protocol: expected piece hash; context canceled; EOF
Jan 7 07:10:53.049151166 (783h41m52.920234212s) [noise=true , address=145.79.227.35:4487 , piece_id=E5X6GYKRPSYHASZXZB62XA3LXB6EJPTMENDRW6BGEDVDUD5BW4TA, seg_pos={0 0}, upload=685546]: Long tail cancellation due to optimal threshold
Jan 7 07:10:53.049299398 (783h41m52.920382434s) [noise=true , address=45.91.116.10:4110 , piece_id=VODTQS7BAXPWLTDDTTW75FXYRDBGM2PSJQTIGCRCRNAMMTE55LJA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123KwusATu1ZKtNNVHZZZBE67ZEnVb8puyfQidBi3fEWREqdf1h): protocol: expected piece hash; context canceled; EOF
Jan 7 07:10:53.049308466 (783h41m52.920391512s) [noise=true , address=45.91.116.10:4110 , piece_id=VODTQS7BAXPWLTDDTTW75FXYRDBGM2PSJQTIGCRCRNAMMTE55LJA, seg_pos={0 0}, upload=685546]: Long tail cancellation due to optimal threshold
Jan 7 07:10:53.049324507 (783h41m52.920407542s) [seg_pos={0 0}, upload=685546]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 7 07:10:53.049388278 (783h41m52.920471313s) [upload=685546]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 7 07:10:53.049392767 (783h41m52.920475812s) [upload=685546]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 7 07:10:53.340338951 (783h41m53.211421996s) [upload=685546]: Done uploading segments...
Jan 7 07:10:53.340359300 (783h41m53.211442335s) [upload=685546]: Upload finished. err: <nil>
Duration: 38.145345301s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [.............X...............X...................X..........] duration:
27.475975645s trace:
https://jaeger.dev.storj.io/trace/36161e151ba11c34
FAIL (2026-01-07 07:16:29 UTC)
orbiter warning: context deadline exceeded
Duration: 27.476018428s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
49 out of 60 is failed (threshold=6) [.XXXXXX.XXXXXXXXXXX.XXXX.XXXX..XXXXX.XXXXX..XXXXX.XXXX.XXXXX] duration:
5.001118501s trace:
https://jaeger.dev.storj.io/trace/51b6a025ca7222db
FAIL (2026-01-07 07:16:18 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001131496s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [......X.XXX.X.X..X....X.X.....X....X.X.....X.......X....XX..] duration:
30.001150689s trace:
https://jaeger.dev.storj.io/trace/3649792573630e65
FAIL (2026-01-07 07:16:05 UTC)
orbiter warning: context deadline exceeded
Duration: 30.001239758s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
49 out of 60 is failed (threshold=6) [X.XXXXXX.XXXXXXXXXXX.XXXX.XXXX..XXXXX.XXXXX..XXXXX.XXXX.XXXX] duration:
5.00168073s trace:
https://jaeger.dev.storj.io/trace/201d875ddf11dcfe
FAIL (2026-01-07 07:15:54 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001705056s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [XX.X...X....X.........X..X.X..X.......XX.X....X....XX....X..] duration:
6.070813268s trace:
https://jaeger.dev.storj.io/trace/566429788081faee
FAIL (2026-01-07 07:15:48 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
) [upload=685582]: Uploading segments...
Jan 7 07:13:32.990108051 (783h44m32.861191096s) [upload=685582]: Got next segment. Inline: false
Jan 7 07:13:32.990479758 (783h44m32.861562803s) [upload=685582]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 7 07:13:32.990486191 (783h44m32.861569226s) [upload=685582]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 7 07:13:33.230730672 (783h44m33.101813717s) [seg_pos={0 0}, upload=685582]: Begin upload segment...
Jan 7 07:13:33.230783873 (783h44m33.101866908s) [seg_pos={0 0}, upload=685582]: Done begin upload segment.
Jan 7 07:13:33.230814070 (783h44m33.101897115s) [noise=true , address=45.91.116.13:4153 , piece_id=27PHIESDJJJKDKJITJODFYQLSF42BX3BVTMOXFJAUGK73D3Z47UA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.230992941 (783h44m33.102075976s) [noise=true , address=45.91.116.5:4300 , piece_id=5FMCKFKIRBUKZGZTTBKCMURBHZTBKQPHEZMHJWJHLM2PLEDFNXSA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.231512250 (783h44m33.102595285s) [noise=true , address=145.79.227.22:4511 , piece_id=VHGXFDPNKXDZI2ARWI7227A5CF3P572VJSQZS75BMMHPAVCPPO7Q, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.231639272 (783h44m33.102722307s) [noise=true , address=145.79.227.4:4185 , piece_id=SMLAUFNVX6M24OXQ2PK2TIP5MXF3KZYRGHBFXYN7OXD5L76PCY3Q, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.231935536 (783h44m33.103018571s) [noise=true , address=145.79.227.31:4798 , piece_id=TKL7NRLI7DYTXU4HEOBJHXQR5CGHU7IG53L2GLENDCORGVVN2KOQ, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.232203125 (783h44m33.10328616s) [noise=true , address=145.79.227.38:4978 , piece_id=QV3BBDP5GVUMTBI2HIE7FWUIUOQVKIOO2D6OJNW2VDCXDFKMISUA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.232580194 (783h44m33.103663229s) [noise=true , address=145.79.227.52:4445 , piece_id=D6QKUQIBGUGV3V2ZELIM6U5UMTHT2765CHQBIKQ2N3BJF3UXH3XA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.232942814 (783h44m33.104025849s) [noise=true , address=45.91.116.8:4047 , piece_id=D6ZXOD5IBAYIVGUWNC6OGIL3OTB7JF6OAFGNGIR2RH7NDWRAEORA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.233250680 (783h44m33.104333725s) [noise=true , address=145.79.227.40:4049 , piece_id=RV43PFYU6UWU5QFHCO7T6F3E3VBNEYQRMA4UCPXT54PP3T3ZD62Q, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.233527197 (783h44m33.104610242s) [noise=true , address=77.90.31.16:4736 , piece_id=6I7NM23KI67OH2RKW4NSGYGJYWZXMH5HVLHLOQFVNCSFW4WZJ5NA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.234521901 (783h44m33.105604936s) [noise=true , address=45.91.116.12:4226 , piece_id=V7NKTZUGNA2HMXA3FSYMIXMSDLQVEUTJB74IW32GYF4XTU45UY6Q, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.235153884 (783h44m33.106236919s) [noise=true , address=145.79.227.45:4231 , piece_id=ESQLZIZB2AVESVJRHVGX4T5AJDEKVGJEJ3JSGA5J25BOQ4UUF7KA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.235615984 (783h44m33.106699019s) [noise=true , address=145.79.227.11:4729 , piece_id=5EQJ3NL4R45GFF35F2WOYWCD77UGR6XUV6B4TIZASZBCBCX6APWA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.236105967 (783h44m33.107189002s) [noise=true , address=145.79.227.5:4221 , piece_id=SHQ2ARJ6XTS7NHVSSNAXW2VR7BPS26MSX76INSRBH6KXPM4SQKEA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.236621799 (783h44m33.107704844s) [noise=true , address=145.79.227.7:4293 , piece_id=WPVYNVVDT4G5DG2HIOLUDVJNIU5O3WFII2H3GIHEZDBST4F2DKGA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.239024364 (783h44m33.110107409s) [noise=true , address=77.90.31.18:4805 , piece_id=YIHVN23OWQTVKXHZIAY4K2C65LU67NA7C6GCC63PXZHCWAZGL7DQ, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:34.837564819 (783h44m34.708647854s) [upload=685582]: Next returned nil segment
Jan 7 07:13:34.837594546 (783h44m34.708677581s) [upload=685582]: Waiting for error group managing segments...
Jan 7 07:13:35.221987466 (783h44m35.093070501s) [noise=true , address=145.79.227.38:4978 , piece_id=QV3BBDP5GVUMTBI2HIE7FWUIUOQVKIOO2D6OJNW2VDCXDFKMISUA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.224032561 (783h44m35.095115596s) [noise=true , address=145.79.227.11:4729 , piece_id=5EQJ3NL4R45GFF35F2WOYWCD77UGR6XUV6B4TIZASZBCBCX6APWA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.224625309 (783h44m35.095708355s) [noise=true , address=145.79.227.52:4445 , piece_id=D6QKUQIBGUGV3V2ZELIM6U5UMTHT2765CHQBIKQ2N3BJF3UXH3XA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.225249578 (783h44m35.096332613s) [noise=true , address=145.79.227.22:4511 , piece_id=VHGXFDPNKXDZI2ARWI7227A5CF3P572VJSQZS75BMMHPAVCPPO7Q, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.233155391 (783h44m35.104238446s) [noise=true , address=145.79.227.45:4231 , piece_id=ESQLZIZB2AVESVJRHVGX4T5AJDEKVGJEJ3JSGA5J25BOQ4UUF7KA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.244418267 (783h44m35.115501302s) [noise=true , address=145.79.227.5:4221 , piece_id=SHQ2ARJ6XTS7NHVSSNAXW2VR7BPS26MSX76INSRBH6KXPM4SQKEA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.253004497 (783h44m35.124087542s) [noise=true , address=145.79.227.31:4798 , piece_id=TKL7NRLI7DYTXU4HEOBJHXQR5CGHU7IG53L2GLENDCORGVVN2KOQ, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.260994110 (783h44m35.132077145s) [noise=true , address=145.79.227.7:4293 , piece_id=WPVYNVVDT4G5DG2HIOLUDVJNIU5O3WFII2H3GIHEZDBST4F2DKGA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.298671554 (783h44m35.169754589s) [noise=true , address=145.79.227.4:4185 , piece_id=SMLAUFNVX6M24OXQ2PK2TIP5MXF3KZYRGHBFXYN7OXD5L76PCY3Q, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.306366076 (783h44m35.177449111s) [noise=true , address=145.79.227.40:4049 , piece_id=RV43PFYU6UWU5QFHCO7T6F3E3VBNEYQRMA4UCPXT54PP3T3ZD62Q, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.519691228 (783h44m35.390774263s) [noise=true , address=77.90.31.16:4736 , piece_id=6I7NM23KI67OH2RKW4NSGYGJYWZXMH5HVLHLOQFVNCSFW4WZJ5NA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.749540043 (783h44m35.620623078s) [noise=true , address=45.91.116.5:4300 , piece_id=5FMCKFKIRBUKZGZTTBKCMURBHZTBKQPHEZMHJWJHLM2PLEDFNXSA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.823582121 (783h44m35.694665156s) [noise=true , address=45.91.116.8:4047 , piece_id=D6ZXOD5IBAYIVGUWNC6OGIL3OTB7JF6OAFGNGIR2RH7NDWRAEORA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.823600715 (783h44m35.69468375s) [seg_pos={0 0}, upload=685582]: Segment reached optimal threshold of 13 pieces.
Jan 7 07:13:35.824004615 (783h44m35.69508765s) [noise=true , address=45.91.116.12:4226 , piece_id=V7NKTZUGNA2HMXA3FSYMIXMSDLQVEUTJB74IW32GYF4XTU45UY6Q, seg_pos={0 0}, upload=685582]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: write tcp 188.166.225.170:54196->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:54196->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54196->45.91.116.12:4226: use of closed network connection
Jan 7 07:13:35.824012139 (783h44m35.695095174s) [noise=true , address=45.91.116.12:4226 , piece_id=V7NKTZUGNA2HMXA3FSYMIXMSDLQVEUTJB74IW32GYF4XTU45UY6Q, seg_pos={0 0}, upload=685582]: Long tail cancellation due to optimal threshold
Jan 7 07:13:35.824197492 (783h44m35.695280527s) [noise=true , address=45.91.116.13:4153 , piece_id=27PHIESDJJJKDKJITJODFYQLSF42BX3BVTMOXFJAUGK73D3Z47UA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: read tcp 188.166.225.170:33886->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 7 07:13:35.824205007 (783h44m35.695288042s) [noise=true , address=45.91.116.13:4153 , piece_id=27PHIESDJJJKDKJITJODFYQLSF42BX3BVTMOXFJAUGK73D3Z47UA, seg_pos={0 0}, upload=685582]: Long tail cancellation due to optimal threshold
Jan 7 07:13:35.824576704 (783h44m35.695659739s) [noise=true , address=77.90.31.18:4805 , piece_id=YIHVN23OWQTVKXHZIAY4K2C65LU67NA7C6GCC63PXZHCWAZGL7DQ, seg_pos={0 0}, upload=685582]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: read tcp 188.166.225.170:38598->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 7 07:13:35.824596592 (783h44m35.695679637s) [noise=true , address=77.90.31.18:4805 , piece_id=YIHVN23OWQTVKXHZIAY4K2C65LU67NA7C6GCC63PXZHCWAZGL7DQ, seg_pos={0 0}, upload=685582]: Long tail cancellation due to optimal threshold
Jan 7 07:13:35.824616139 (783h44m35.695699184s) [seg_pos={0 0}, upload=685582]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 7 07:13:35.824682215 (783h44m35.69576526s) [upload=685582]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 7 07:13:35.824686843 (783h44m35.695769878s) [upload=685582]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 7 07:13:36.105700746 (783h44m35.976783791s) [upload=685582]: Done uploading segments...
Jan 7 07:13:36.105723649 (783h44m35.976806684s) [upload=685582]: Upload finished. err: <nil>
Duration: 6.070840521s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
49 out of 60 is failed (threshold=6) [XX.XXXXXX.XXXXXXXXXXX.XXXX.XXXX..XXXXX.XXXXX..XXXXX.XXXX.XXX] duration:
5.000454916s trace:
https://jaeger.dev.storj.io/trace/3f74d18bdb526f41
FAIL (2026-01-07 07:15:37 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000469354s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
3 out of 60 is failed (threshold=3) [..............X...............X...................X.........] duration:
32.813731826s trace:
https://jaeger.dev.storj.io/trace/36161e151ba11c34
FAIL (2026-01-07 07:15:29 UTC)
orbiter warning: context deadline exceeded
Duration: 32.813759829s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
49 out of 60 is failed (threshold=6) [XXX.XXXXXX.XXXXXXXXXXX.XXXX.XXXX..XXXXX.XXXXX..XXXXX.XXXX.XX] duration:
5.001225064s trace:
https://jaeger.dev.storj.io/trace/280f6ccab72bf94b
FAIL (2026-01-07 07:15:18 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001248487s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [...X........................................X........X......] duration:
6.396100477s trace:
https://jaeger.dev.storj.io/trace/1ea01aa868761fdc
FAIL (2026-01-07 07:15:15 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
133194519 (783h41m50.004277565s) [noise=true , address=145.79.227.32:4851 , piece_id=5W74ZJOGSKONFPVTJJPHDWNFHLTWRU4KEVX25X4PQCKTRJZR3CZA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.133698059 (783h41m50.004781104s) [noise=true , address=145.79.227.33:4897 , piece_id=NLJJNXY2DGHMZWGJRSKGTYDVYZLYZJSVYVIZ35IUCMSI34RD5TBA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.134246032 (783h41m50.005329067s) [noise=true , address=145.79.227.44:4227 , piece_id=TCERBF6EX3JHAV63JYLUYA54OHYY33YEM25EQ6CIRS4746KNCULA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.134617359 (783h41m50.005700404s) [noise=true , address=145.79.227.30:4795 , piece_id=G7EGUQIODKUJDE6FEWO5B5C3RQYQ37SAHQYC6BI7X4JS6A7VMOXQ, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.135018602 (783h41m50.006101637s) [noise=true , address=77.90.31.16:4741 , piece_id=G4TPC522KLMJBYHYW2SYTUMJFVP4K54R63ZIHOCPQINQO5WE7Q2Q, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.135492385 (783h41m50.00657542s) [noise=true , address=145.79.227.41:4092 , piece_id=3WQEAVSBFU5KF6WVNGACEVQ6M42JEMWW6OHCXPCXVNZCOG4XSMGQ, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.135895753 (783h41m50.006978799s) [noise=true , address=145.79.227.34:4926 , piece_id=AUBI2PYIJK5YTKFZHL5WF7BQOKNKEPSTZQBERRGEOFZN7RE7T2SA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.136238677 (783h41m50.007321722s) [noise=true , address=193.149.164.143:4699 , piece_id=DFHC3CPJBT2U7MW5HSNGIGTPMD3QZD4ON62UJ2HNNVWRXOJ4A6PQ, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.136546201 (783h41m50.007629236s) [noise=true , address=145.79.227.36:4543 , piece_id=MUK6THUJD4QHK4VLL3R6DSZP47VV5RJAUPKG6RTPR7F6GTIC6GIA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.136565809 (783h41m50.007648844s) [noise=true , address=145.79.227.29:4946 , piece_id=FR3BLE7ANKJYAQG5GZJ4RL4543FQ55K7VHSJMQGCLC7UDRNKCASA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.137150974 (783h41m50.008234019s) [noise=true , address=145.79.227.35:4487 , piece_id=E5X6GYKRPSYHASZXZB62XA3LXB6EJPTMENDRW6BGEDVDUD5BW4TA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.137656586 (783h41m50.008739621s) [noise=true , address=77.90.31.11:4584 , piece_id=LIDDXUURY5ARSGQ6ETUFION36NJFWMIGJHOSOIE4ZUXJPTJNCGSA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.138273391 (783h41m50.009356436s) [noise=true , address=64.49.13.169:4713 , piece_id=6JKGRVL7KQOZVIIJM3OFQUHWUEF42KMW7Y6I5EPVKVRTKEQVBFNQ, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.138867362 (783h41m50.009950407s) [noise=true , address=64.49.13.155:4916 , piece_id=FPLKLM4RLSO54V7XX3YZF7CGXWSGKNBAKEYQ5FYFYHVE7CTS6PIA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.139440193 (783h41m50.010523238s) [noise=true , address=145.79.227.49:4372 , piece_id=HQ535NWA7OLGPNMYWPMHDJL3ICULP4GEXMYTKLDTC2SPMQYCRZOQ, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.139988797 (783h41m50.011071832s) [noise=true , address=45.91.116.10:4110 , piece_id=VODTQS7BAXPWLTDDTTW75FXYRDBGM2PSJQTIGCRCRNAMMTE55LJA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.140847663 (783h41m50.011930698s) [noise=true , address=45.91.116.14:4200 , piece_id=ON5HNVYRNJTJCNV4JU6NI2ITS2VKZNNMYOBZUZQ7OLXPYAUQL2JQ, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:51.568606354 (783h41m51.439689389s) [upload=685546]: Next returned nil segment
Jan 7 07:10:51.568621312 (783h41m51.439704347s) [upload=685546]: Waiting for error group managing segments...
Jan 7 07:10:51.990916323 (783h41m51.861999368s) [noise=true , address=64.49.13.155:4916 , piece_id=FPLKLM4RLSO54V7XX3YZF7CGXWSGKNBAKEYQ5FYFYHVE7CTS6PIA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:51.994944574 (783h41m51.866027619s) [noise=true , address=145.79.227.44:4227 , piece_id=TCERBF6EX3JHAV63JYLUYA54OHYY33YEM25EQ6CIRS4746KNCULA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.000551040 (783h41m51.871634105s) [noise=true , address=145.79.227.41:4092 , piece_id=3WQEAVSBFU5KF6WVNGACEVQ6M42JEMWW6OHCXPCXVNZCOG4XSMGQ, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.005752454 (783h41m51.876835509s) [noise=true , address=145.79.227.30:4795 , piece_id=G7EGUQIODKUJDE6FEWO5B5C3RQYQ37SAHQYC6BI7X4JS6A7VMOXQ, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.006180148 (783h41m51.877263203s) [noise=true , address=145.79.227.32:4851 , piece_id=5W74ZJOGSKONFPVTJJPHDWNFHLTWRU4KEVX25X4PQCKTRJZR3CZA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.062554132 (783h41m51.933637177s) [noise=true , address=64.49.13.169:4713 , piece_id=6JKGRVL7KQOZVIIJM3OFQUHWUEF42KMW7Y6I5EPVKVRTKEQVBFNQ, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.070238635 (783h41m51.94132167s) [noise=true , address=145.79.227.34:4926 , piece_id=AUBI2PYIJK5YTKFZHL5WF7BQOKNKEPSTZQBERRGEOFZN7RE7T2SA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.088972997 (783h41m51.960056032s) [noise=true , address=145.79.227.33:4897 , piece_id=NLJJNXY2DGHMZWGJRSKGTYDVYZLYZJSVYVIZ35IUCMSI34RD5TBA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.281595960 (783h41m52.152678995s) [noise=true , address=77.90.31.12:4603 , piece_id=HPBI4VUKMT2KFKRW2JTX32HB2YGDGLB2V4BFRVVQ2725Y2IC2LOA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.356241606 (783h41m52.227324641s) [noise=true , address=77.90.31.11:4584 , piece_id=LIDDXUURY5ARSGQ6ETUFION36NJFWMIGJHOSOIE4ZUXJPTJNCGSA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.679275149 (783h41m52.550358204s) [noise=true , address=45.91.116.14:4200 , piece_id=ON5HNVYRNJTJCNV4JU6NI2ITS2VKZNNMYOBZUZQ7OLXPYAUQL2JQ, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.848093704 (783h41m52.719176749s) [noise=true , address=193.149.164.143:4699 , piece_id=DFHC3CPJBT2U7MW5HSNGIGTPMD3QZD4ON62UJ2HNNVWRXOJ4A6PQ, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:53.022732271 (783h41m52.893815316s) [noise=true , address=145.79.227.49:4372 , piece_id=HQ535NWA7OLGPNMYWPMHDJL3ICULP4GEXMYTKLDTC2SPMQYCRZOQ, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:53.048522630 (783h41m52.919605675s) [noise=true , address=145.79.227.36:4543 , piece_id=MUK6THUJD4QHK4VLL3R6DSZP47VV5RJAUPKG6RTPR7F6GTIC6GIA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:53.048547036 (783h41m52.919630082s) [seg_pos={0 0}, upload=685546]: Segment reached optimal threshold of 14 pieces.
Jan 7 07:10:53.048761935 (783h41m52.919844971s) [noise=true , address=77.90.31.16:4741 , piece_id=G4TPC522KLMJBYHYW2SYTUMJFVP4K54R63ZIHOCPQINQO5WE7Q2Q, seg_pos={0 0}, upload=685546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR): protocol: expected piece hash; context canceled; EOF
Jan 7 07:10:53.048771714 (783h41m52.919854749s) [noise=true , address=77.90.31.16:4741 , piece_id=G4TPC522KLMJBYHYW2SYTUMJFVP4K54R63ZIHOCPQINQO5WE7Q2Q, seg_pos={0 0}, upload=685546]: Long tail cancellation due to optimal threshold
Jan 7 07:10:53.049042570 (783h41m52.920125605s) [noise=true , address=145.79.227.29:4946 , piece_id=FR3BLE7ANKJYAQG5GZJ4RL4543FQ55K7VHSJMQGCLC7UDRNKCASA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12v9Q7aigjWPNXyshHmQt9YKrDu5af1vd1TxkDeHeKixSrWgrLX): protocol: expected piece hash; context canceled; EOF
Jan 7 07:10:53.049049483 (783h41m52.920132518s) [noise=true , address=145.79.227.29:4946 , piece_id=FR3BLE7ANKJYAQG5GZJ4RL4543FQ55K7VHSJMQGCLC7UDRNKCASA, seg_pos={0 0}, upload=685546]: Long tail cancellation due to optimal threshold
Jan 7 07:10:53.049146277 (783h41m52.920229322s) [noise=true , address=145.79.227.35:4487 , piece_id=E5X6GYKRPSYHASZXZB62XA3LXB6EJPTMENDRW6BGEDVDUD5BW4TA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AdZLDV7UBbV2DimCHzmoNxycLigvLMx6Fy9U9QB35WTtiV1KX): protocol: expected piece hash; context canceled; EOF
Jan 7 07:10:53.049151166 (783h41m52.920234212s) [noise=true , address=145.79.227.35:4487 , piece_id=E5X6GYKRPSYHASZXZB62XA3LXB6EJPTMENDRW6BGEDVDUD5BW4TA, seg_pos={0 0}, upload=685546]: Long tail cancellation due to optimal threshold
Jan 7 07:10:53.049299398 (783h41m52.920382434s) [noise=true , address=45.91.116.10:4110 , piece_id=VODTQS7BAXPWLTDDTTW75FXYRDBGM2PSJQTIGCRCRNAMMTE55LJA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123KwusATu1ZKtNNVHZZZBE67ZEnVb8puyfQidBi3fEWREqdf1h): protocol: expected piece hash; context canceled; EOF
Jan 7 07:10:53.049308466 (783h41m52.920391512s) [noise=true , address=45.91.116.10:4110 , piece_id=VODTQS7BAXPWLTDDTTW75FXYRDBGM2PSJQTIGCRCRNAMMTE55LJA, seg_pos={0 0}, upload=685546]: Long tail cancellation due to optimal threshold
Jan 7 07:10:53.049324507 (783h41m52.920407542s) [seg_pos={0 0}, upload=685546]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 7 07:10:53.049388278 (783h41m52.920471313s) [upload=685546]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 7 07:10:53.049392767 (783h41m52.920475812s) [upload=685546]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 7 07:10:53.340338951 (783h41m53.211421996s) [upload=685546]: Done uploading segments...
Jan 7 07:10:53.340359300 (783h41m53.211442335s) [upload=685546]: Upload finished. err: <nil>
Duration: 6.396118762s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [.......X.XXX.X.X..X....X.X.....X....X.X.....X.......X....XX.] duration:
22.5821929s trace:
https://jaeger.dev.storj.io/trace/3649792573630e65
FAIL (2026-01-07 07:14:48 UTC)
orbiter warning: context deadline exceeded
Duration: 22.582219561s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [X..............X...............X...................X........] duration:
7.962611622s trace:
https://jaeger.dev.storj.io/trace/36161e151ba11c34
FAIL (2026-01-07 07:14:43 UTC)
orbiter warning: context deadline exceeded
Duration: 7.962637381s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [.XX.X...X....X.........X..X.X..X.......XX.X....X....XX....X.] duration:
6.025329236s trace:
https://jaeger.dev.storj.io/trace/566429788081faee
FAIL (2026-01-07 07:14:34 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
) [upload=685582]: Uploading segments...
Jan 7 07:13:32.990108051 (783h44m32.861191096s) [upload=685582]: Got next segment. Inline: false
Jan 7 07:13:32.990479758 (783h44m32.861562803s) [upload=685582]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 7 07:13:32.990486191 (783h44m32.861569226s) [upload=685582]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 7 07:13:33.230730672 (783h44m33.101813717s) [seg_pos={0 0}, upload=685582]: Begin upload segment...
Jan 7 07:13:33.230783873 (783h44m33.101866908s) [seg_pos={0 0}, upload=685582]: Done begin upload segment.
Jan 7 07:13:33.230814070 (783h44m33.101897115s) [noise=true , address=45.91.116.13:4153 , piece_id=27PHIESDJJJKDKJITJODFYQLSF42BX3BVTMOXFJAUGK73D3Z47UA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.230992941 (783h44m33.102075976s) [noise=true , address=45.91.116.5:4300 , piece_id=5FMCKFKIRBUKZGZTTBKCMURBHZTBKQPHEZMHJWJHLM2PLEDFNXSA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.231512250 (783h44m33.102595285s) [noise=true , address=145.79.227.22:4511 , piece_id=VHGXFDPNKXDZI2ARWI7227A5CF3P572VJSQZS75BMMHPAVCPPO7Q, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.231639272 (783h44m33.102722307s) [noise=true , address=145.79.227.4:4185 , piece_id=SMLAUFNVX6M24OXQ2PK2TIP5MXF3KZYRGHBFXYN7OXD5L76PCY3Q, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.231935536 (783h44m33.103018571s) [noise=true , address=145.79.227.31:4798 , piece_id=TKL7NRLI7DYTXU4HEOBJHXQR5CGHU7IG53L2GLENDCORGVVN2KOQ, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.232203125 (783h44m33.10328616s) [noise=true , address=145.79.227.38:4978 , piece_id=QV3BBDP5GVUMTBI2HIE7FWUIUOQVKIOO2D6OJNW2VDCXDFKMISUA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.232580194 (783h44m33.103663229s) [noise=true , address=145.79.227.52:4445 , piece_id=D6QKUQIBGUGV3V2ZELIM6U5UMTHT2765CHQBIKQ2N3BJF3UXH3XA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.232942814 (783h44m33.104025849s) [noise=true , address=45.91.116.8:4047 , piece_id=D6ZXOD5IBAYIVGUWNC6OGIL3OTB7JF6OAFGNGIR2RH7NDWRAEORA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.233250680 (783h44m33.104333725s) [noise=true , address=145.79.227.40:4049 , piece_id=RV43PFYU6UWU5QFHCO7T6F3E3VBNEYQRMA4UCPXT54PP3T3ZD62Q, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.233527197 (783h44m33.104610242s) [noise=true , address=77.90.31.16:4736 , piece_id=6I7NM23KI67OH2RKW4NSGYGJYWZXMH5HVLHLOQFVNCSFW4WZJ5NA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.234521901 (783h44m33.105604936s) [noise=true , address=45.91.116.12:4226 , piece_id=V7NKTZUGNA2HMXA3FSYMIXMSDLQVEUTJB74IW32GYF4XTU45UY6Q, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.235153884 (783h44m33.106236919s) [noise=true , address=145.79.227.45:4231 , piece_id=ESQLZIZB2AVESVJRHVGX4T5AJDEKVGJEJ3JSGA5J25BOQ4UUF7KA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.235615984 (783h44m33.106699019s) [noise=true , address=145.79.227.11:4729 , piece_id=5EQJ3NL4R45GFF35F2WOYWCD77UGR6XUV6B4TIZASZBCBCX6APWA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.236105967 (783h44m33.107189002s) [noise=true , address=145.79.227.5:4221 , piece_id=SHQ2ARJ6XTS7NHVSSNAXW2VR7BPS26MSX76INSRBH6KXPM4SQKEA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.236621799 (783h44m33.107704844s) [noise=true , address=145.79.227.7:4293 , piece_id=WPVYNVVDT4G5DG2HIOLUDVJNIU5O3WFII2H3GIHEZDBST4F2DKGA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.239024364 (783h44m33.110107409s) [noise=true , address=77.90.31.18:4805 , piece_id=YIHVN23OWQTVKXHZIAY4K2C65LU67NA7C6GCC63PXZHCWAZGL7DQ, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:34.837564819 (783h44m34.708647854s) [upload=685582]: Next returned nil segment
Jan 7 07:13:34.837594546 (783h44m34.708677581s) [upload=685582]: Waiting for error group managing segments...
Jan 7 07:13:35.221987466 (783h44m35.093070501s) [noise=true , address=145.79.227.38:4978 , piece_id=QV3BBDP5GVUMTBI2HIE7FWUIUOQVKIOO2D6OJNW2VDCXDFKMISUA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.224032561 (783h44m35.095115596s) [noise=true , address=145.79.227.11:4729 , piece_id=5EQJ3NL4R45GFF35F2WOYWCD77UGR6XUV6B4TIZASZBCBCX6APWA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.224625309 (783h44m35.095708355s) [noise=true , address=145.79.227.52:4445 , piece_id=D6QKUQIBGUGV3V2ZELIM6U5UMTHT2765CHQBIKQ2N3BJF3UXH3XA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.225249578 (783h44m35.096332613s) [noise=true , address=145.79.227.22:4511 , piece_id=VHGXFDPNKXDZI2ARWI7227A5CF3P572VJSQZS75BMMHPAVCPPO7Q, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.233155391 (783h44m35.104238446s) [noise=true , address=145.79.227.45:4231 , piece_id=ESQLZIZB2AVESVJRHVGX4T5AJDEKVGJEJ3JSGA5J25BOQ4UUF7KA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.244418267 (783h44m35.115501302s) [noise=true , address=145.79.227.5:4221 , piece_id=SHQ2ARJ6XTS7NHVSSNAXW2VR7BPS26MSX76INSRBH6KXPM4SQKEA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.253004497 (783h44m35.124087542s) [noise=true , address=145.79.227.31:4798 , piece_id=TKL7NRLI7DYTXU4HEOBJHXQR5CGHU7IG53L2GLENDCORGVVN2KOQ, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.260994110 (783h44m35.132077145s) [noise=true , address=145.79.227.7:4293 , piece_id=WPVYNVVDT4G5DG2HIOLUDVJNIU5O3WFII2H3GIHEZDBST4F2DKGA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.298671554 (783h44m35.169754589s) [noise=true , address=145.79.227.4:4185 , piece_id=SMLAUFNVX6M24OXQ2PK2TIP5MXF3KZYRGHBFXYN7OXD5L76PCY3Q, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.306366076 (783h44m35.177449111s) [noise=true , address=145.79.227.40:4049 , piece_id=RV43PFYU6UWU5QFHCO7T6F3E3VBNEYQRMA4UCPXT54PP3T3ZD62Q, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.519691228 (783h44m35.390774263s) [noise=true , address=77.90.31.16:4736 , piece_id=6I7NM23KI67OH2RKW4NSGYGJYWZXMH5HVLHLOQFVNCSFW4WZJ5NA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.749540043 (783h44m35.620623078s) [noise=true , address=45.91.116.5:4300 , piece_id=5FMCKFKIRBUKZGZTTBKCMURBHZTBKQPHEZMHJWJHLM2PLEDFNXSA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.823582121 (783h44m35.694665156s) [noise=true , address=45.91.116.8:4047 , piece_id=D6ZXOD5IBAYIVGUWNC6OGIL3OTB7JF6OAFGNGIR2RH7NDWRAEORA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.823600715 (783h44m35.69468375s) [seg_pos={0 0}, upload=685582]: Segment reached optimal threshold of 13 pieces.
Jan 7 07:13:35.824004615 (783h44m35.69508765s) [noise=true , address=45.91.116.12:4226 , piece_id=V7NKTZUGNA2HMXA3FSYMIXMSDLQVEUTJB74IW32GYF4XTU45UY6Q, seg_pos={0 0}, upload=685582]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: write tcp 188.166.225.170:54196->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:54196->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54196->45.91.116.12:4226: use of closed network connection
Jan 7 07:13:35.824012139 (783h44m35.695095174s) [noise=true , address=45.91.116.12:4226 , piece_id=V7NKTZUGNA2HMXA3FSYMIXMSDLQVEUTJB74IW32GYF4XTU45UY6Q, seg_pos={0 0}, upload=685582]: Long tail cancellation due to optimal threshold
Jan 7 07:13:35.824197492 (783h44m35.695280527s) [noise=true , address=45.91.116.13:4153 , piece_id=27PHIESDJJJKDKJITJODFYQLSF42BX3BVTMOXFJAUGK73D3Z47UA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: read tcp 188.166.225.170:33886->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 7 07:13:35.824205007 (783h44m35.695288042s) [noise=true , address=45.91.116.13:4153 , piece_id=27PHIESDJJJKDKJITJODFYQLSF42BX3BVTMOXFJAUGK73D3Z47UA, seg_pos={0 0}, upload=685582]: Long tail cancellation due to optimal threshold
Jan 7 07:13:35.824576704 (783h44m35.695659739s) [noise=true , address=77.90.31.18:4805 , piece_id=YIHVN23OWQTVKXHZIAY4K2C65LU67NA7C6GCC63PXZHCWAZGL7DQ, seg_pos={0 0}, upload=685582]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: read tcp 188.166.225.170:38598->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 7 07:13:35.824596592 (783h44m35.695679637s) [noise=true , address=77.90.31.18:4805 , piece_id=YIHVN23OWQTVKXHZIAY4K2C65LU67NA7C6GCC63PXZHCWAZGL7DQ, seg_pos={0 0}, upload=685582]: Long tail cancellation due to optimal threshold
Jan 7 07:13:35.824616139 (783h44m35.695699184s) [seg_pos={0 0}, upload=685582]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 7 07:13:35.824682215 (783h44m35.69576526s) [upload=685582]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 7 07:13:35.824686843 (783h44m35.695769878s) [upload=685582]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 7 07:13:36.105700746 (783h44m35.976783791s) [upload=685582]: Done uploading segments...
Jan 7 07:13:36.105723649 (783h44m35.976806684s) [upload=685582]: Upload finished. err: <nil>
Duration: 6.02535709s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [....X........................................X........X.....] duration:
6.431619866s trace:
https://jaeger.dev.storj.io/trace/1ea01aa868761fdc
FAIL (2026-01-07 07:14:26 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
133194519 (783h41m50.004277565s) [noise=true , address=145.79.227.32:4851 , piece_id=5W74ZJOGSKONFPVTJJPHDWNFHLTWRU4KEVX25X4PQCKTRJZR3CZA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.133698059 (783h41m50.004781104s) [noise=true , address=145.79.227.33:4897 , piece_id=NLJJNXY2DGHMZWGJRSKGTYDVYZLYZJSVYVIZ35IUCMSI34RD5TBA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.134246032 (783h41m50.005329067s) [noise=true , address=145.79.227.44:4227 , piece_id=TCERBF6EX3JHAV63JYLUYA54OHYY33YEM25EQ6CIRS4746KNCULA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.134617359 (783h41m50.005700404s) [noise=true , address=145.79.227.30:4795 , piece_id=G7EGUQIODKUJDE6FEWO5B5C3RQYQ37SAHQYC6BI7X4JS6A7VMOXQ, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.135018602 (783h41m50.006101637s) [noise=true , address=77.90.31.16:4741 , piece_id=G4TPC522KLMJBYHYW2SYTUMJFVP4K54R63ZIHOCPQINQO5WE7Q2Q, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.135492385 (783h41m50.00657542s) [noise=true , address=145.79.227.41:4092 , piece_id=3WQEAVSBFU5KF6WVNGACEVQ6M42JEMWW6OHCXPCXVNZCOG4XSMGQ, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.135895753 (783h41m50.006978799s) [noise=true , address=145.79.227.34:4926 , piece_id=AUBI2PYIJK5YTKFZHL5WF7BQOKNKEPSTZQBERRGEOFZN7RE7T2SA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.136238677 (783h41m50.007321722s) [noise=true , address=193.149.164.143:4699 , piece_id=DFHC3CPJBT2U7MW5HSNGIGTPMD3QZD4ON62UJ2HNNVWRXOJ4A6PQ, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.136546201 (783h41m50.007629236s) [noise=true , address=145.79.227.36:4543 , piece_id=MUK6THUJD4QHK4VLL3R6DSZP47VV5RJAUPKG6RTPR7F6GTIC6GIA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.136565809 (783h41m50.007648844s) [noise=true , address=145.79.227.29:4946 , piece_id=FR3BLE7ANKJYAQG5GZJ4RL4543FQ55K7VHSJMQGCLC7UDRNKCASA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.137150974 (783h41m50.008234019s) [noise=true , address=145.79.227.35:4487 , piece_id=E5X6GYKRPSYHASZXZB62XA3LXB6EJPTMENDRW6BGEDVDUD5BW4TA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.137656586 (783h41m50.008739621s) [noise=true , address=77.90.31.11:4584 , piece_id=LIDDXUURY5ARSGQ6ETUFION36NJFWMIGJHOSOIE4ZUXJPTJNCGSA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.138273391 (783h41m50.009356436s) [noise=true , address=64.49.13.169:4713 , piece_id=6JKGRVL7KQOZVIIJM3OFQUHWUEF42KMW7Y6I5EPVKVRTKEQVBFNQ, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.138867362 (783h41m50.009950407s) [noise=true , address=64.49.13.155:4916 , piece_id=FPLKLM4RLSO54V7XX3YZF7CGXWSGKNBAKEYQ5FYFYHVE7CTS6PIA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.139440193 (783h41m50.010523238s) [noise=true , address=145.79.227.49:4372 , piece_id=HQ535NWA7OLGPNMYWPMHDJL3ICULP4GEXMYTKLDTC2SPMQYCRZOQ, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.139988797 (783h41m50.011071832s) [noise=true , address=45.91.116.10:4110 , piece_id=VODTQS7BAXPWLTDDTTW75FXYRDBGM2PSJQTIGCRCRNAMMTE55LJA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.140847663 (783h41m50.011930698s) [noise=true , address=45.91.116.14:4200 , piece_id=ON5HNVYRNJTJCNV4JU6NI2ITS2VKZNNMYOBZUZQ7OLXPYAUQL2JQ, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:51.568606354 (783h41m51.439689389s) [upload=685546]: Next returned nil segment
Jan 7 07:10:51.568621312 (783h41m51.439704347s) [upload=685546]: Waiting for error group managing segments...
Jan 7 07:10:51.990916323 (783h41m51.861999368s) [noise=true , address=64.49.13.155:4916 , piece_id=FPLKLM4RLSO54V7XX3YZF7CGXWSGKNBAKEYQ5FYFYHVE7CTS6PIA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:51.994944574 (783h41m51.866027619s) [noise=true , address=145.79.227.44:4227 , piece_id=TCERBF6EX3JHAV63JYLUYA54OHYY33YEM25EQ6CIRS4746KNCULA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.000551040 (783h41m51.871634105s) [noise=true , address=145.79.227.41:4092 , piece_id=3WQEAVSBFU5KF6WVNGACEVQ6M42JEMWW6OHCXPCXVNZCOG4XSMGQ, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.005752454 (783h41m51.876835509s) [noise=true , address=145.79.227.30:4795 , piece_id=G7EGUQIODKUJDE6FEWO5B5C3RQYQ37SAHQYC6BI7X4JS6A7VMOXQ, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.006180148 (783h41m51.877263203s) [noise=true , address=145.79.227.32:4851 , piece_id=5W74ZJOGSKONFPVTJJPHDWNFHLTWRU4KEVX25X4PQCKTRJZR3CZA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.062554132 (783h41m51.933637177s) [noise=true , address=64.49.13.169:4713 , piece_id=6JKGRVL7KQOZVIIJM3OFQUHWUEF42KMW7Y6I5EPVKVRTKEQVBFNQ, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.070238635 (783h41m51.94132167s) [noise=true , address=145.79.227.34:4926 , piece_id=AUBI2PYIJK5YTKFZHL5WF7BQOKNKEPSTZQBERRGEOFZN7RE7T2SA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.088972997 (783h41m51.960056032s) [noise=true , address=145.79.227.33:4897 , piece_id=NLJJNXY2DGHMZWGJRSKGTYDVYZLYZJSVYVIZ35IUCMSI34RD5TBA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.281595960 (783h41m52.152678995s) [noise=true , address=77.90.31.12:4603 , piece_id=HPBI4VUKMT2KFKRW2JTX32HB2YGDGLB2V4BFRVVQ2725Y2IC2LOA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.356241606 (783h41m52.227324641s) [noise=true , address=77.90.31.11:4584 , piece_id=LIDDXUURY5ARSGQ6ETUFION36NJFWMIGJHOSOIE4ZUXJPTJNCGSA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.679275149 (783h41m52.550358204s) [noise=true , address=45.91.116.14:4200 , piece_id=ON5HNVYRNJTJCNV4JU6NI2ITS2VKZNNMYOBZUZQ7OLXPYAUQL2JQ, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.848093704 (783h41m52.719176749s) [noise=true , address=193.149.164.143:4699 , piece_id=DFHC3CPJBT2U7MW5HSNGIGTPMD3QZD4ON62UJ2HNNVWRXOJ4A6PQ, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:53.022732271 (783h41m52.893815316s) [noise=true , address=145.79.227.49:4372 , piece_id=HQ535NWA7OLGPNMYWPMHDJL3ICULP4GEXMYTKLDTC2SPMQYCRZOQ, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:53.048522630 (783h41m52.919605675s) [noise=true , address=145.79.227.36:4543 , piece_id=MUK6THUJD4QHK4VLL3R6DSZP47VV5RJAUPKG6RTPR7F6GTIC6GIA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:53.048547036 (783h41m52.919630082s) [seg_pos={0 0}, upload=685546]: Segment reached optimal threshold of 14 pieces.
Jan 7 07:10:53.048761935 (783h41m52.919844971s) [noise=true , address=77.90.31.16:4741 , piece_id=G4TPC522KLMJBYHYW2SYTUMJFVP4K54R63ZIHOCPQINQO5WE7Q2Q, seg_pos={0 0}, upload=685546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR): protocol: expected piece hash; context canceled; EOF
Jan 7 07:10:53.048771714 (783h41m52.919854749s) [noise=true , address=77.90.31.16:4741 , piece_id=G4TPC522KLMJBYHYW2SYTUMJFVP4K54R63ZIHOCPQINQO5WE7Q2Q, seg_pos={0 0}, upload=685546]: Long tail cancellation due to optimal threshold
Jan 7 07:10:53.049042570 (783h41m52.920125605s) [noise=true , address=145.79.227.29:4946 , piece_id=FR3BLE7ANKJYAQG5GZJ4RL4543FQ55K7VHSJMQGCLC7UDRNKCASA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12v9Q7aigjWPNXyshHmQt9YKrDu5af1vd1TxkDeHeKixSrWgrLX): protocol: expected piece hash; context canceled; EOF
Jan 7 07:10:53.049049483 (783h41m52.920132518s) [noise=true , address=145.79.227.29:4946 , piece_id=FR3BLE7ANKJYAQG5GZJ4RL4543FQ55K7VHSJMQGCLC7UDRNKCASA, seg_pos={0 0}, upload=685546]: Long tail cancellation due to optimal threshold
Jan 7 07:10:53.049146277 (783h41m52.920229322s) [noise=true , address=145.79.227.35:4487 , piece_id=E5X6GYKRPSYHASZXZB62XA3LXB6EJPTMENDRW6BGEDVDUD5BW4TA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AdZLDV7UBbV2DimCHzmoNxycLigvLMx6Fy9U9QB35WTtiV1KX): protocol: expected piece hash; context canceled; EOF
Jan 7 07:10:53.049151166 (783h41m52.920234212s) [noise=true , address=145.79.227.35:4487 , piece_id=E5X6GYKRPSYHASZXZB62XA3LXB6EJPTMENDRW6BGEDVDUD5BW4TA, seg_pos={0 0}, upload=685546]: Long tail cancellation due to optimal threshold
Jan 7 07:10:53.049299398 (783h41m52.920382434s) [noise=true , address=45.91.116.10:4110 , piece_id=VODTQS7BAXPWLTDDTTW75FXYRDBGM2PSJQTIGCRCRNAMMTE55LJA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123KwusATu1ZKtNNVHZZZBE67ZEnVb8puyfQidBi3fEWREqdf1h): protocol: expected piece hash; context canceled; EOF
Jan 7 07:10:53.049308466 (783h41m52.920391512s) [noise=true , address=45.91.116.10:4110 , piece_id=VODTQS7BAXPWLTDDTTW75FXYRDBGM2PSJQTIGCRCRNAMMTE55LJA, seg_pos={0 0}, upload=685546]: Long tail cancellation due to optimal threshold
Jan 7 07:10:53.049324507 (783h41m52.920407542s) [seg_pos={0 0}, upload=685546]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 7 07:10:53.049388278 (783h41m52.920471313s) [upload=685546]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 7 07:10:53.049392767 (783h41m52.920475812s) [upload=685546]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 7 07:10:53.340338951 (783h41m53.211421996s) [upload=685546]: Done uploading segments...
Jan 7 07:10:53.340359300 (783h41m53.211442335s) [upload=685546]: Upload finished. err: <nil>
Duration: 6.431645724s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [.X..............X...............X...................X.......] duration:
15.085793631s trace:
https://jaeger.dev.storj.io/trace/36161e151ba11c34
FAIL (2026-01-07 07:14:24 UTC)
orbiter warning: context deadline exceeded
Duration: 15.08624556s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
49 out of 60 is failed (threshold=6) [XXXX.XXXXXX.XXXXXXXXXXX.XXXX.XXXX..XXXXX.XXXXX..XXXXX.XXXX.X] duration:
5.001475891s trace:
https://jaeger.dev.storj.io/trace/1fa1f050306ec5ea
FAIL (2026-01-07 07:14:09 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001502211s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [........X.XXX.X.X..X....X.X.....X....X.X.....X.......X....XX] duration:
30.001609575s trace:
https://jaeger.dev.storj.io/trace/3649792573630e65
FAIL (2026-01-07 07:14:08 UTC)
orbiter warning: context deadline exceeded
Duration: 30.001632267s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [..X..............X...............X...................X......] duration:
19.827722539s trace:
https://jaeger.dev.storj.io/trace/36161e151ba11c34
FAIL (2026-01-07 07:14:04 UTC)
orbiter warning: context deadline exceeded
Duration: 19.827748407s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [..XX.X...X....X.........X..X.X..X.......XX.X....X....XX....X] duration:
30.00375387s trace:
https://jaeger.dev.storj.io/trace/566429788081faee
FAIL (2026-01-07 07:14:02 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
) [upload=685582]: Uploading segments...
Jan 7 07:13:32.990108051 (783h44m32.861191096s) [upload=685582]: Got next segment. Inline: false
Jan 7 07:13:32.990479758 (783h44m32.861562803s) [upload=685582]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 7 07:13:32.990486191 (783h44m32.861569226s) [upload=685582]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 7 07:13:33.230730672 (783h44m33.101813717s) [seg_pos={0 0}, upload=685582]: Begin upload segment...
Jan 7 07:13:33.230783873 (783h44m33.101866908s) [seg_pos={0 0}, upload=685582]: Done begin upload segment.
Jan 7 07:13:33.230814070 (783h44m33.101897115s) [noise=true , address=45.91.116.13:4153 , piece_id=27PHIESDJJJKDKJITJODFYQLSF42BX3BVTMOXFJAUGK73D3Z47UA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.230992941 (783h44m33.102075976s) [noise=true , address=45.91.116.5:4300 , piece_id=5FMCKFKIRBUKZGZTTBKCMURBHZTBKQPHEZMHJWJHLM2PLEDFNXSA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.231512250 (783h44m33.102595285s) [noise=true , address=145.79.227.22:4511 , piece_id=VHGXFDPNKXDZI2ARWI7227A5CF3P572VJSQZS75BMMHPAVCPPO7Q, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.231639272 (783h44m33.102722307s) [noise=true , address=145.79.227.4:4185 , piece_id=SMLAUFNVX6M24OXQ2PK2TIP5MXF3KZYRGHBFXYN7OXD5L76PCY3Q, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.231935536 (783h44m33.103018571s) [noise=true , address=145.79.227.31:4798 , piece_id=TKL7NRLI7DYTXU4HEOBJHXQR5CGHU7IG53L2GLENDCORGVVN2KOQ, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.232203125 (783h44m33.10328616s) [noise=true , address=145.79.227.38:4978 , piece_id=QV3BBDP5GVUMTBI2HIE7FWUIUOQVKIOO2D6OJNW2VDCXDFKMISUA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.232580194 (783h44m33.103663229s) [noise=true , address=145.79.227.52:4445 , piece_id=D6QKUQIBGUGV3V2ZELIM6U5UMTHT2765CHQBIKQ2N3BJF3UXH3XA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.232942814 (783h44m33.104025849s) [noise=true , address=45.91.116.8:4047 , piece_id=D6ZXOD5IBAYIVGUWNC6OGIL3OTB7JF6OAFGNGIR2RH7NDWRAEORA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.233250680 (783h44m33.104333725s) [noise=true , address=145.79.227.40:4049 , piece_id=RV43PFYU6UWU5QFHCO7T6F3E3VBNEYQRMA4UCPXT54PP3T3ZD62Q, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.233527197 (783h44m33.104610242s) [noise=true , address=77.90.31.16:4736 , piece_id=6I7NM23KI67OH2RKW4NSGYGJYWZXMH5HVLHLOQFVNCSFW4WZJ5NA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.234521901 (783h44m33.105604936s) [noise=true , address=45.91.116.12:4226 , piece_id=V7NKTZUGNA2HMXA3FSYMIXMSDLQVEUTJB74IW32GYF4XTU45UY6Q, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.235153884 (783h44m33.106236919s) [noise=true , address=145.79.227.45:4231 , piece_id=ESQLZIZB2AVESVJRHVGX4T5AJDEKVGJEJ3JSGA5J25BOQ4UUF7KA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.235615984 (783h44m33.106699019s) [noise=true , address=145.79.227.11:4729 , piece_id=5EQJ3NL4R45GFF35F2WOYWCD77UGR6XUV6B4TIZASZBCBCX6APWA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.236105967 (783h44m33.107189002s) [noise=true , address=145.79.227.5:4221 , piece_id=SHQ2ARJ6XTS7NHVSSNAXW2VR7BPS26MSX76INSRBH6KXPM4SQKEA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.236621799 (783h44m33.107704844s) [noise=true , address=145.79.227.7:4293 , piece_id=WPVYNVVDT4G5DG2HIOLUDVJNIU5O3WFII2H3GIHEZDBST4F2DKGA, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:33.239024364 (783h44m33.110107409s) [noise=true , address=77.90.31.18:4805 , piece_id=YIHVN23OWQTVKXHZIAY4K2C65LU67NA7C6GCC63PXZHCWAZGL7DQ, seg_pos={0 0}, upload=685582]: Uploading piece...
Jan 7 07:13:34.837564819 (783h44m34.708647854s) [upload=685582]: Next returned nil segment
Jan 7 07:13:34.837594546 (783h44m34.708677581s) [upload=685582]: Waiting for error group managing segments...
Jan 7 07:13:35.221987466 (783h44m35.093070501s) [noise=true , address=145.79.227.38:4978 , piece_id=QV3BBDP5GVUMTBI2HIE7FWUIUOQVKIOO2D6OJNW2VDCXDFKMISUA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.224032561 (783h44m35.095115596s) [noise=true , address=145.79.227.11:4729 , piece_id=5EQJ3NL4R45GFF35F2WOYWCD77UGR6XUV6B4TIZASZBCBCX6APWA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.224625309 (783h44m35.095708355s) [noise=true , address=145.79.227.52:4445 , piece_id=D6QKUQIBGUGV3V2ZELIM6U5UMTHT2765CHQBIKQ2N3BJF3UXH3XA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.225249578 (783h44m35.096332613s) [noise=true , address=145.79.227.22:4511 , piece_id=VHGXFDPNKXDZI2ARWI7227A5CF3P572VJSQZS75BMMHPAVCPPO7Q, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.233155391 (783h44m35.104238446s) [noise=true , address=145.79.227.45:4231 , piece_id=ESQLZIZB2AVESVJRHVGX4T5AJDEKVGJEJ3JSGA5J25BOQ4UUF7KA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.244418267 (783h44m35.115501302s) [noise=true , address=145.79.227.5:4221 , piece_id=SHQ2ARJ6XTS7NHVSSNAXW2VR7BPS26MSX76INSRBH6KXPM4SQKEA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.253004497 (783h44m35.124087542s) [noise=true , address=145.79.227.31:4798 , piece_id=TKL7NRLI7DYTXU4HEOBJHXQR5CGHU7IG53L2GLENDCORGVVN2KOQ, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.260994110 (783h44m35.132077145s) [noise=true , address=145.79.227.7:4293 , piece_id=WPVYNVVDT4G5DG2HIOLUDVJNIU5O3WFII2H3GIHEZDBST4F2DKGA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.298671554 (783h44m35.169754589s) [noise=true , address=145.79.227.4:4185 , piece_id=SMLAUFNVX6M24OXQ2PK2TIP5MXF3KZYRGHBFXYN7OXD5L76PCY3Q, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.306366076 (783h44m35.177449111s) [noise=true , address=145.79.227.40:4049 , piece_id=RV43PFYU6UWU5QFHCO7T6F3E3VBNEYQRMA4UCPXT54PP3T3ZD62Q, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.519691228 (783h44m35.390774263s) [noise=true , address=77.90.31.16:4736 , piece_id=6I7NM23KI67OH2RKW4NSGYGJYWZXMH5HVLHLOQFVNCSFW4WZJ5NA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.749540043 (783h44m35.620623078s) [noise=true , address=45.91.116.5:4300 , piece_id=5FMCKFKIRBUKZGZTTBKCMURBHZTBKQPHEZMHJWJHLM2PLEDFNXSA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.823582121 (783h44m35.694665156s) [noise=true , address=45.91.116.8:4047 , piece_id=D6ZXOD5IBAYIVGUWNC6OGIL3OTB7JF6OAFGNGIR2RH7NDWRAEORA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: <nil>
Jan 7 07:13:35.823600715 (783h44m35.69468375s) [seg_pos={0 0}, upload=685582]: Segment reached optimal threshold of 13 pieces.
Jan 7 07:13:35.824004615 (783h44m35.69508765s) [noise=true , address=45.91.116.12:4226 , piece_id=V7NKTZUGNA2HMXA3FSYMIXMSDLQVEUTJB74IW32GYF4XTU45UY6Q, seg_pos={0 0}, upload=685582]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: write tcp 188.166.225.170:54196->45.91.116.12:4226: use of closed network connection; write tcp 188.166.225.170:54196->45.91.116.12:4226: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54196->45.91.116.12:4226: use of closed network connection
Jan 7 07:13:35.824012139 (783h44m35.695095174s) [noise=true , address=45.91.116.12:4226 , piece_id=V7NKTZUGNA2HMXA3FSYMIXMSDLQVEUTJB74IW32GYF4XTU45UY6Q, seg_pos={0 0}, upload=685582]: Long tail cancellation due to optimal threshold
Jan 7 07:13:35.824197492 (783h44m35.695280527s) [noise=true , address=45.91.116.13:4153 , piece_id=27PHIESDJJJKDKJITJODFYQLSF42BX3BVTMOXFJAUGK73D3Z47UA, seg_pos={0 0}, upload=685582]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: read tcp 188.166.225.170:33886->45.91.116.13:4153: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 7 07:13:35.824205007 (783h44m35.695288042s) [noise=true , address=45.91.116.13:4153 , piece_id=27PHIESDJJJKDKJITJODFYQLSF42BX3BVTMOXFJAUGK73D3Z47UA, seg_pos={0 0}, upload=685582]: Long tail cancellation due to optimal threshold
Jan 7 07:13:35.824576704 (783h44m35.695659739s) [noise=true , address=77.90.31.18:4805 , piece_id=YIHVN23OWQTVKXHZIAY4K2C65LU67NA7C6GCC63PXZHCWAZGL7DQ, seg_pos={0 0}, upload=685582]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT): protocol: read tcp 188.166.225.170:38598->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Jan 7 07:13:35.824596592 (783h44m35.695679637s) [noise=true , address=77.90.31.18:4805 , piece_id=YIHVN23OWQTVKXHZIAY4K2C65LU67NA7C6GCC63PXZHCWAZGL7DQ, seg_pos={0 0}, upload=685582]: Long tail cancellation due to optimal threshold
Jan 7 07:13:35.824616139 (783h44m35.695699184s) [seg_pos={0 0}, upload=685582]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 7 07:13:35.824682215 (783h44m35.69576526s) [upload=685582]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 7 07:13:35.824686843 (783h44m35.695769878s) [upload=685582]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 7 07:13:36.105700746 (783h44m35.976783791s) [upload=685582]: Done uploading segments...
Jan 7 07:13:36.105723649 (783h44m35.976806684s) [upload=685582]: Upload finished. err: <nil>
Duration: 30.003803977s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [.....X........................................X........X....] duration:
15.538473346s trace:
https://jaeger.dev.storj.io/trace/1ea01aa868761fdc
FAIL (2026-01-07 07:13:50 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
133194519 (783h41m50.004277565s) [noise=true , address=145.79.227.32:4851 , piece_id=5W74ZJOGSKONFPVTJJPHDWNFHLTWRU4KEVX25X4PQCKTRJZR3CZA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.133698059 (783h41m50.004781104s) [noise=true , address=145.79.227.33:4897 , piece_id=NLJJNXY2DGHMZWGJRSKGTYDVYZLYZJSVYVIZ35IUCMSI34RD5TBA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.134246032 (783h41m50.005329067s) [noise=true , address=145.79.227.44:4227 , piece_id=TCERBF6EX3JHAV63JYLUYA54OHYY33YEM25EQ6CIRS4746KNCULA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.134617359 (783h41m50.005700404s) [noise=true , address=145.79.227.30:4795 , piece_id=G7EGUQIODKUJDE6FEWO5B5C3RQYQ37SAHQYC6BI7X4JS6A7VMOXQ, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.135018602 (783h41m50.006101637s) [noise=true , address=77.90.31.16:4741 , piece_id=G4TPC522KLMJBYHYW2SYTUMJFVP4K54R63ZIHOCPQINQO5WE7Q2Q, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.135492385 (783h41m50.00657542s) [noise=true , address=145.79.227.41:4092 , piece_id=3WQEAVSBFU5KF6WVNGACEVQ6M42JEMWW6OHCXPCXVNZCOG4XSMGQ, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.135895753 (783h41m50.006978799s) [noise=true , address=145.79.227.34:4926 , piece_id=AUBI2PYIJK5YTKFZHL5WF7BQOKNKEPSTZQBERRGEOFZN7RE7T2SA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.136238677 (783h41m50.007321722s) [noise=true , address=193.149.164.143:4699 , piece_id=DFHC3CPJBT2U7MW5HSNGIGTPMD3QZD4ON62UJ2HNNVWRXOJ4A6PQ, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.136546201 (783h41m50.007629236s) [noise=true , address=145.79.227.36:4543 , piece_id=MUK6THUJD4QHK4VLL3R6DSZP47VV5RJAUPKG6RTPR7F6GTIC6GIA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.136565809 (783h41m50.007648844s) [noise=true , address=145.79.227.29:4946 , piece_id=FR3BLE7ANKJYAQG5GZJ4RL4543FQ55K7VHSJMQGCLC7UDRNKCASA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.137150974 (783h41m50.008234019s) [noise=true , address=145.79.227.35:4487 , piece_id=E5X6GYKRPSYHASZXZB62XA3LXB6EJPTMENDRW6BGEDVDUD5BW4TA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.137656586 (783h41m50.008739621s) [noise=true , address=77.90.31.11:4584 , piece_id=LIDDXUURY5ARSGQ6ETUFION36NJFWMIGJHOSOIE4ZUXJPTJNCGSA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.138273391 (783h41m50.009356436s) [noise=true , address=64.49.13.169:4713 , piece_id=6JKGRVL7KQOZVIIJM3OFQUHWUEF42KMW7Y6I5EPVKVRTKEQVBFNQ, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.138867362 (783h41m50.009950407s) [noise=true , address=64.49.13.155:4916 , piece_id=FPLKLM4RLSO54V7XX3YZF7CGXWSGKNBAKEYQ5FYFYHVE7CTS6PIA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.139440193 (783h41m50.010523238s) [noise=true , address=145.79.227.49:4372 , piece_id=HQ535NWA7OLGPNMYWPMHDJL3ICULP4GEXMYTKLDTC2SPMQYCRZOQ, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.139988797 (783h41m50.011071832s) [noise=true , address=45.91.116.10:4110 , piece_id=VODTQS7BAXPWLTDDTTW75FXYRDBGM2PSJQTIGCRCRNAMMTE55LJA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.140847663 (783h41m50.011930698s) [noise=true , address=45.91.116.14:4200 , piece_id=ON5HNVYRNJTJCNV4JU6NI2ITS2VKZNNMYOBZUZQ7OLXPYAUQL2JQ, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:51.568606354 (783h41m51.439689389s) [upload=685546]: Next returned nil segment
Jan 7 07:10:51.568621312 (783h41m51.439704347s) [upload=685546]: Waiting for error group managing segments...
Jan 7 07:10:51.990916323 (783h41m51.861999368s) [noise=true , address=64.49.13.155:4916 , piece_id=FPLKLM4RLSO54V7XX3YZF7CGXWSGKNBAKEYQ5FYFYHVE7CTS6PIA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:51.994944574 (783h41m51.866027619s) [noise=true , address=145.79.227.44:4227 , piece_id=TCERBF6EX3JHAV63JYLUYA54OHYY33YEM25EQ6CIRS4746KNCULA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.000551040 (783h41m51.871634105s) [noise=true , address=145.79.227.41:4092 , piece_id=3WQEAVSBFU5KF6WVNGACEVQ6M42JEMWW6OHCXPCXVNZCOG4XSMGQ, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.005752454 (783h41m51.876835509s) [noise=true , address=145.79.227.30:4795 , piece_id=G7EGUQIODKUJDE6FEWO5B5C3RQYQ37SAHQYC6BI7X4JS6A7VMOXQ, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.006180148 (783h41m51.877263203s) [noise=true , address=145.79.227.32:4851 , piece_id=5W74ZJOGSKONFPVTJJPHDWNFHLTWRU4KEVX25X4PQCKTRJZR3CZA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.062554132 (783h41m51.933637177s) [noise=true , address=64.49.13.169:4713 , piece_id=6JKGRVL7KQOZVIIJM3OFQUHWUEF42KMW7Y6I5EPVKVRTKEQVBFNQ, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.070238635 (783h41m51.94132167s) [noise=true , address=145.79.227.34:4926 , piece_id=AUBI2PYIJK5YTKFZHL5WF7BQOKNKEPSTZQBERRGEOFZN7RE7T2SA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.088972997 (783h41m51.960056032s) [noise=true , address=145.79.227.33:4897 , piece_id=NLJJNXY2DGHMZWGJRSKGTYDVYZLYZJSVYVIZ35IUCMSI34RD5TBA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.281595960 (783h41m52.152678995s) [noise=true , address=77.90.31.12:4603 , piece_id=HPBI4VUKMT2KFKRW2JTX32HB2YGDGLB2V4BFRVVQ2725Y2IC2LOA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.356241606 (783h41m52.227324641s) [noise=true , address=77.90.31.11:4584 , piece_id=LIDDXUURY5ARSGQ6ETUFION36NJFWMIGJHOSOIE4ZUXJPTJNCGSA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.679275149 (783h41m52.550358204s) [noise=true , address=45.91.116.14:4200 , piece_id=ON5HNVYRNJTJCNV4JU6NI2ITS2VKZNNMYOBZUZQ7OLXPYAUQL2JQ, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.848093704 (783h41m52.719176749s) [noise=true , address=193.149.164.143:4699 , piece_id=DFHC3CPJBT2U7MW5HSNGIGTPMD3QZD4ON62UJ2HNNVWRXOJ4A6PQ, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:53.022732271 (783h41m52.893815316s) [noise=true , address=145.79.227.49:4372 , piece_id=HQ535NWA7OLGPNMYWPMHDJL3ICULP4GEXMYTKLDTC2SPMQYCRZOQ, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:53.048522630 (783h41m52.919605675s) [noise=true , address=145.79.227.36:4543 , piece_id=MUK6THUJD4QHK4VLL3R6DSZP47VV5RJAUPKG6RTPR7F6GTIC6GIA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:53.048547036 (783h41m52.919630082s) [seg_pos={0 0}, upload=685546]: Segment reached optimal threshold of 14 pieces.
Jan 7 07:10:53.048761935 (783h41m52.919844971s) [noise=true , address=77.90.31.16:4741 , piece_id=G4TPC522KLMJBYHYW2SYTUMJFVP4K54R63ZIHOCPQINQO5WE7Q2Q, seg_pos={0 0}, upload=685546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR): protocol: expected piece hash; context canceled; EOF
Jan 7 07:10:53.048771714 (783h41m52.919854749s) [noise=true , address=77.90.31.16:4741 , piece_id=G4TPC522KLMJBYHYW2SYTUMJFVP4K54R63ZIHOCPQINQO5WE7Q2Q, seg_pos={0 0}, upload=685546]: Long tail cancellation due to optimal threshold
Jan 7 07:10:53.049042570 (783h41m52.920125605s) [noise=true , address=145.79.227.29:4946 , piece_id=FR3BLE7ANKJYAQG5GZJ4RL4543FQ55K7VHSJMQGCLC7UDRNKCASA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12v9Q7aigjWPNXyshHmQt9YKrDu5af1vd1TxkDeHeKixSrWgrLX): protocol: expected piece hash; context canceled; EOF
Jan 7 07:10:53.049049483 (783h41m52.920132518s) [noise=true , address=145.79.227.29:4946 , piece_id=FR3BLE7ANKJYAQG5GZJ4RL4543FQ55K7VHSJMQGCLC7UDRNKCASA, seg_pos={0 0}, upload=685546]: Long tail cancellation due to optimal threshold
Jan 7 07:10:53.049146277 (783h41m52.920229322s) [noise=true , address=145.79.227.35:4487 , piece_id=E5X6GYKRPSYHASZXZB62XA3LXB6EJPTMENDRW6BGEDVDUD5BW4TA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AdZLDV7UBbV2DimCHzmoNxycLigvLMx6Fy9U9QB35WTtiV1KX): protocol: expected piece hash; context canceled; EOF
Jan 7 07:10:53.049151166 (783h41m52.920234212s) [noise=true , address=145.79.227.35:4487 , piece_id=E5X6GYKRPSYHASZXZB62XA3LXB6EJPTMENDRW6BGEDVDUD5BW4TA, seg_pos={0 0}, upload=685546]: Long tail cancellation due to optimal threshold
Jan 7 07:10:53.049299398 (783h41m52.920382434s) [noise=true , address=45.91.116.10:4110 , piece_id=VODTQS7BAXPWLTDDTTW75FXYRDBGM2PSJQTIGCRCRNAMMTE55LJA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123KwusATu1ZKtNNVHZZZBE67ZEnVb8puyfQidBi3fEWREqdf1h): protocol: expected piece hash; context canceled; EOF
Jan 7 07:10:53.049308466 (783h41m52.920391512s) [noise=true , address=45.91.116.10:4110 , piece_id=VODTQS7BAXPWLTDDTTW75FXYRDBGM2PSJQTIGCRCRNAMMTE55LJA, seg_pos={0 0}, upload=685546]: Long tail cancellation due to optimal threshold
Jan 7 07:10:53.049324507 (783h41m52.920407542s) [seg_pos={0 0}, upload=685546]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 7 07:10:53.049388278 (783h41m52.920471313s) [upload=685546]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 7 07:10:53.049392767 (783h41m52.920475812s) [upload=685546]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 7 07:10:53.340338951 (783h41m53.211421996s) [upload=685546]: Done uploading segments...
Jan 7 07:10:53.340359300 (783h41m53.211442335s) [upload=685546]: Upload finished. err: <nil>
Duration: 15.538490409s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
49 out of 60 is failed (threshold=6) [XXXXX.XXXXXX.XXXXXXXXXXX.XXXX.XXXX..XXXXX.XXXXX..XXXXX.XXXX.] duration:
4.186910699s trace:
https://jaeger.dev.storj.io/trace/57a51158fdbdd198
FAIL (2026-01-07 07:13:09 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.186934655s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
15 out of 60 is failed (threshold=3) [.........X.XXX.X.X..X....X.X.....X....X.X.....X.......X....X] duration:
30.001234787s trace:
https://jaeger.dev.storj.io/trace/140e9098feaa3265
FAIL (2026-01-07 07:13:07 UTC)
orbiter warning: context deadline exceeded
Duration: 30.001254804s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [...X..............X...............X...................X.....] duration:
20.676228104s trace:
https://jaeger.dev.storj.io/trace/36161e151ba11c34
FAIL (2026-01-07 07:13:02 UTC)
orbiter warning: context deadline exceeded
Duration: 20.676248104s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [X..XX.X...X....X.........X..X.X..X.......XX.X....X....XX....] duration:
5.772565519s trace:
https://jaeger.dev.storj.io/trace/50ae9c48f3048fc3
FAIL (2026-01-07 07:12:45 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: SOAAPUTTLH3GV2TF2HUAN4BOA7OZKF4QICOHY4S5QEKESZD5N24A) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: BJHBHMTYA5HFEAKMPMM2PSNFBI7X32H2LP6VRY753L32SKGUKCSQ) context deadline exceeded
======LOGS======
Jan 7 07:10:31.248040589 (783h41m31.119123644s) [upload=685543]: Starting upload
Jan 7 07:10:31.248154456 (783h41m31.119237511s) [upload=685543]: Done starting upload
Jan 7 07:10:31.248536263 (783h41m31.119619308s) [upload=685543]: Uploading segments...
Jan 7 07:10:31.248606457 (783h41m31.119689502s) [upload=685543]: Got next segment. Inline: false
Jan 7 07:10:31.248647264 (783h41m31.119730309s) [upload=685543]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 7 07:10:31.248652915 (783h41m31.11973596s) [upload=685543]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 7 07:10:31.492628981 (783h41m31.363712026s) [seg_pos={0 0}, upload=685543]: Begin upload segment...
Jan 7 07:10:31.492693132 (783h41m31.363776177s) [seg_pos={0 0}, upload=685543]: Done begin upload segment.
Jan 7 07:10:31.492797992 (783h41m31.363881027s) [noise=true , address=145.79.227.45:4230 , piece_id=SQFIEICI2DR5H6UQM3D2JITSFG7NCZNBUMDVOUFNXPEAVYYG3UFQ, seg_pos={0 0}, upload=685543]: Uploading piece...
Jan 7 07:10:31.493578138 (783h41m31.364661183s) [noise=true , address=45.91.116.11:4117 , piece_id=Z5XRYSZPRFNSGBJCX66NZADXGM67M5Y7JDPMUWAS3KGFRZ3YTT4Q, seg_pos={0 0}, upload=685543]: Uploading piece...
Jan 7 07:10:31.494131151 (783h41m31.365214186s) [noise=true , address=145.79.227.37:4565 , piece_id=CPOFNSLYHUHQ4JCQMJEX24ZOBPJJCZYZZOAOSEREU4YDCIMLHEOQ, seg_pos={0 0}, upload=685543]: Uploading piece...
Jan 7 07:10:31.494488501 (783h41m31.365571546s) [noise=true , address=77.90.31.19:4841 , piece_id=T4ZU4XYKFBN4Q4FCHIWPFGQKYUBRZFUY2LTNP57EB4RPIR3LCLWQ, seg_pos={0 0}, upload=685543]: Uploading piece...
Jan 7 07:10:31.494776870 (783h41m31.365859915s) [noise=true , address=145.79.227.44:4195 , piece_id=DTUIBJ4JAWQGUOYOVRSMZEHQHBSZHYLISBYXKS22GEHDT7PO2CLA, seg_pos={0 0}, upload=685543]: Uploading piece...
Jan 7 07:10:31.495074447 (783h41m31.366157482s) [noise=true , address=145.79.227.12:4767 , piece_id=SRD2DTDKANNQNF4Q6H6D5QPYJU4SGL42ANBIDDB4AEICFTO6XERQ, seg_pos={0 0}, upload=685543]: Uploading piece...
Jan 7 07:10:31.495142487 (783h41m31.366225532s) [noise=true , address=77.90.31.16:4736 , piece_id=SOAAPUTTLH3GV2TF2HUAN4BOA7OZKF4QICOHY4S5QEKESZD5N24A, seg_pos={0 0}, upload=685543]: Uploading piece...
Jan 7 07:10:31.495649812 (783h41m31.366732847s) [noise=true , address=145.79.227.40:4050 , piece_id=EFGM7CQSPJJRM2ASQWDO6BUDKZ2KQBEGNIHLN4GNQ6SUWKVEEZ3Q, seg_pos={0 0}, upload=685543]: Uploading piece...
Jan 7 07:10:31.496148402 (783h41m31.367231447s) [noise=true , address=145.79.227.31:4799 , piece_id=BCSP53DP7YKGDPF5K37PZJQKD6TN7225D5RBFWOUO7OP4SBHMFBQ, seg_pos={0 0}, upload=685543]: Uploading piece...
Jan 7 07:10:31.496702267 (783h41m31.367785312s) [noise=true , address=77.90.31.11:4554 , piece_id=APWWSHCAFJKZEF5H3FDSJOXMXKKHNJEHO7CRXKYU3B3X4VDMBRZA, seg_pos={0 0}, upload=685543]: Uploading piece...
Jan 7 07:10:31.497256683 (783h41m31.368339728s) [noise=true , address=77.90.31.13:4629 , piece_id=TZJJYNZBZP5CNNLEC6TQHBN4XU4WQKMAHAPFJAUF5H2WTOK2DDCQ, seg_pos={0 0}, upload=685543]: Uploading piece...
Jan 7 07:10:31.499267872 (783h41m31.370350917s) [noise=true , address=145.79.227.26:4654 , piece_id=ODJVMCWVJGMYISGDRP4TLR4ABXXD5ZNUXKOSKDHV2HGVQLNKGD6Q, seg_pos={0 0}, upload=685543]: Uploading piece...
Jan 7 07:10:31.500996664 (783h41m31.372079709s) [noise=true , address=145.79.227.42:4121 , piece_id=RDI73ICBIBVAVGJABHW7TQRSUD6OYAMFZQJM7JLU2N7UT44OCSQQ, seg_pos={0 0}, upload=685543]: Uploading piece...
Jan 7 07:10:31.502637298 (783h41m31.373720333s) [noise=true , address=45.91.116.7:4010 , piece_id=BJHBHMTYA5HFEAKMPMM2PSNFBI7X32H2LP6VRY753L32SKGUKCSQ, seg_pos={0 0}, upload=685543]: Uploading piece...
Jan 7 07:10:31.504268755 (783h41m31.3753518s) [noise=true , address=145.79.227.6:4258 , piece_id=QWU6TB6IVDSS6H5AG4IMHOXPALH2AYZSEM36GTHY5KA6XPKCBXXQ, seg_pos={0 0}, upload=685543]: Uploading piece...
Jan 7 07:10:31.506042783 (783h41m31.377125818s) [noise=true , address=145.79.227.2:4113 , piece_id=YQ54OQ26JK34YPLN3TZXR5LWJOFN3RUVZJ5YUZXGI2RY2OKLRFSA, seg_pos={0 0}, upload=685543]: Uploading piece...
Jan 7 07:10:33.202789827 (783h41m33.073872872s) [upload=685543]: Next returned nil segment
Jan 7 07:10:33.202822870 (783h41m33.073905905s) [upload=685543]: Waiting for error group managing segments...
Jan 7 07:10:33.484280988 (783h41m33.355364023s) [noise=true , address=145.79.227.44:4195 , piece_id=DTUIBJ4JAWQGUOYOVRSMZEHQHBSZHYLISBYXKS22GEHDT7PO2CLA, seg_pos={0 0}, upload=685543]: Done uploading piece. err: <nil>
Jan 7 07:10:33.488021811 (783h41m33.359104846s) [noise=true , address=145.79.227.12:4767 , piece_id=SRD2DTDKANNQNF4Q6H6D5QPYJU4SGL42ANBIDDB4AEICFTO6XERQ, seg_pos={0 0}, upload=685543]: Done uploading piece. err: <nil>
Jan 7 07:10:33.491589184 (783h41m33.362672239s) [noise=true , address=145.79.227.40:4050 , piece_id=EFGM7CQSPJJRM2ASQWDO6BUDKZ2KQBEGNIHLN4GNQ6SUWKVEEZ3Q, seg_pos={0 0}, upload=685543]: Done uploading piece. err: <nil>
Jan 7 07:10:33.498955099 (783h41m33.370038134s) [noise=true , address=145.79.227.6:4258 , piece_id=QWU6TB6IVDSS6H5AG4IMHOXPALH2AYZSEM36GTHY5KA6XPKCBXXQ, seg_pos={0 0}, upload=685543]: Done uploading piece. err: <nil>
Jan 7 07:10:33.501171139 (783h41m33.372254174s) [noise=true , address=145.79.227.37:4565 , piece_id=CPOFNSLYHUHQ4JCQMJEX24ZOBPJJCZYZZOAOSEREU4YDCIMLHEOQ, seg_pos={0 0}, upload=685543]: Done uploading piece. err: <nil>
Jan 7 07:10:33.510816175 (783h41m33.38189922s) [noise=true , address=145.79.227.31:4799 , piece_id=BCSP53DP7YKGDPF5K37PZJQKD6TN7225D5RBFWOUO7OP4SBHMFBQ, seg_pos={0 0}, upload=685543]: Done uploading piece. err: <nil>
Jan 7 07:10:33.511084094 (783h41m33.382167129s) [noise=true , address=145.79.227.45:4230 , piece_id=SQFIEICI2DR5H6UQM3D2JITSFG7NCZNBUMDVOUFNXPEAVYYG3UFQ, seg_pos={0 0}, upload=685543]: Done uploading piece. err: <nil>
Jan 7 07:10:33.516566915 (783h41m33.38764997s) [noise=true , address=145.79.227.26:4654 , piece_id=ODJVMCWVJGMYISGDRP4TLR4ABXXD5ZNUXKOSKDHV2HGVQLNKGD6Q, seg_pos={0 0}, upload=685543]: Done uploading piece. err: <nil>
Jan 7 07:10:33.517421321 (783h41m33.388504357s) [noise=true , address=145.79.227.42:4121 , piece_id=RDI73ICBIBVAVGJABHW7TQRSUD6OYAMFZQJM7JLU2N7UT44OCSQQ, seg_pos={0 0}, upload=685543]: Done uploading piece. err: <nil>
Jan 7 07:10:33.518568827 (783h41m33.389651872s) [noise=true , address=145.79.227.2:4113 , piece_id=YQ54OQ26JK34YPLN3TZXR5LWJOFN3RUVZJ5YUZXGI2RY2OKLRFSA, seg_pos={0 0}, upload=685543]: Done uploading piece. err: <nil>
Jan 7 07:10:33.781941837 (783h41m33.653024882s) [noise=true , address=77.90.31.16:4736 , piece_id=SOAAPUTTLH3GV2TF2HUAN4BOA7OZKF4QICOHY4S5QEKESZD5N24A, seg_pos={0 0}, upload=685543]: Done uploading piece. err: <nil>
Jan 7 07:10:33.851158897 (783h41m33.722241932s) [noise=true , address=77.90.31.13:4629 , piece_id=TZJJYNZBZP5CNNLEC6TQHBN4XU4WQKMAHAPFJAUF5H2WTOK2DDCQ, seg_pos={0 0}, upload=685543]: Done uploading piece. err: <nil>
Jan 7 07:10:33.985266949 (783h41m33.856349984s) [noise=true , address=45.91.116.7:4010 , piece_id=BJHBHMTYA5HFEAKMPMM2PSNFBI7X32H2LP6VRY753L32SKGUKCSQ, seg_pos={0 0}, upload=685543]: Done uploading piece. err: <nil>
Jan 7 07:10:33.985294601 (783h41m33.856377647s) [seg_pos={0 0}, upload=685543]: Segment reached optimal threshold of 13 pieces.
Jan 7 07:10:33.985506545 (783h41m33.85658959s) [noise=true , address=77.90.31.11:4554 , piece_id=APWWSHCAFJKZEF5H3FDSJOXMXKKHNJEHO7CRXKYU3B3X4VDMBRZA, seg_pos={0 0}, upload=685543]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: write tcp 188.166.225.170:43140->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:43140->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43140->77.90.31.11:4554: use of closed network connection
Jan 7 07:10:33.985515993 (783h41m33.856599028s) [noise=true , address=77.90.31.11:4554 , piece_id=APWWSHCAFJKZEF5H3FDSJOXMXKKHNJEHO7CRXKYU3B3X4VDMBRZA, seg_pos={0 0}, upload=685543]: Long tail cancellation due to optimal threshold
Jan 7 07:10:33.985592638 (783h41m33.856675673s) [noise=true , address=77.90.31.19:4841 , piece_id=T4ZU4XYKFBN4Q4FCHIWPFGQKYUBRZFUY2LTNP57EB4RPIR3LCLWQ, seg_pos={0 0}, upload=685543]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF
Jan 7 07:10:33.985599121 (783h41m33.856682156s) [noise=true , address=77.90.31.19:4841 , piece_id=T4ZU4XYKFBN4Q4FCHIWPFGQKYUBRZFUY2LTNP57EB4RPIR3LCLWQ, seg_pos={0 0}, upload=685543]: Long tail cancellation due to optimal threshold
Jan 7 07:10:33.985754036 (783h41m33.856837071s) [noise=true , address=45.91.116.11:4117 , piece_id=Z5XRYSZPRFNSGBJCX66NZADXGM67M5Y7JDPMUWAS3KGFRZ3YTT4Q, seg_pos={0 0}, upload=685543]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF
Jan 7 07:10:33.985759727 (783h41m33.856842762s) [noise=true , address=45.91.116.11:4117 , piece_id=Z5XRYSZPRFNSGBJCX66NZADXGM67M5Y7JDPMUWAS3KGFRZ3YTT4Q, seg_pos={0 0}, upload=685543]: Long tail cancellation due to optimal threshold
Jan 7 07:10:33.985872070 (783h41m33.856955106s) [seg_pos={0 0}, upload=685543]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 7 07:10:33.985962032 (783h41m33.857045077s) [upload=685543]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 7 07:10:33.985966591 (783h41m33.857049636s) [upload=685543]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 7 07:10:34.275584985 (783h41m34.14666802s) [upload=685543]: Done uploading segments...
Jan 7 07:10:34.275603481 (783h41m34.146686516s) [upload=685543]: Upload finished. err: <nil>
Duration: 5.772593983s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
49 out of 60 is failed (threshold=6) [.XXXXX.XXXXXX.XXXXXXXXXXX.XXXX.XXXX..XXXXX.XXXXX..XXXXX.XXXX] duration:
5.001535283s trace:
https://jaeger.dev.storj.io/trace/57a51158fdbdd198
FAIL (2026-01-07 07:12:40 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001557506s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
4 out of 60 is failed (threshold=3) [....X..............X...............X...................X....] duration:
7.172763013s trace:
https://jaeger.dev.storj.io/trace/36161e151ba11c34
FAIL (2026-01-07 07:12:30 UTC)
orbiter warning: context deadline exceeded
Duration: 7.172790756s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [......X........................................X........X...] duration:
6.537440716s trace:
https://jaeger.dev.storj.io/trace/1ea01aa868761fdc
FAIL (2026-01-07 07:12:29 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
133194519 (783h41m50.004277565s) [noise=true , address=145.79.227.32:4851 , piece_id=5W74ZJOGSKONFPVTJJPHDWNFHLTWRU4KEVX25X4PQCKTRJZR3CZA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.133698059 (783h41m50.004781104s) [noise=true , address=145.79.227.33:4897 , piece_id=NLJJNXY2DGHMZWGJRSKGTYDVYZLYZJSVYVIZ35IUCMSI34RD5TBA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.134246032 (783h41m50.005329067s) [noise=true , address=145.79.227.44:4227 , piece_id=TCERBF6EX3JHAV63JYLUYA54OHYY33YEM25EQ6CIRS4746KNCULA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.134617359 (783h41m50.005700404s) [noise=true , address=145.79.227.30:4795 , piece_id=G7EGUQIODKUJDE6FEWO5B5C3RQYQ37SAHQYC6BI7X4JS6A7VMOXQ, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.135018602 (783h41m50.006101637s) [noise=true , address=77.90.31.16:4741 , piece_id=G4TPC522KLMJBYHYW2SYTUMJFVP4K54R63ZIHOCPQINQO5WE7Q2Q, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.135492385 (783h41m50.00657542s) [noise=true , address=145.79.227.41:4092 , piece_id=3WQEAVSBFU5KF6WVNGACEVQ6M42JEMWW6OHCXPCXVNZCOG4XSMGQ, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.135895753 (783h41m50.006978799s) [noise=true , address=145.79.227.34:4926 , piece_id=AUBI2PYIJK5YTKFZHL5WF7BQOKNKEPSTZQBERRGEOFZN7RE7T2SA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.136238677 (783h41m50.007321722s) [noise=true , address=193.149.164.143:4699 , piece_id=DFHC3CPJBT2U7MW5HSNGIGTPMD3QZD4ON62UJ2HNNVWRXOJ4A6PQ, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.136546201 (783h41m50.007629236s) [noise=true , address=145.79.227.36:4543 , piece_id=MUK6THUJD4QHK4VLL3R6DSZP47VV5RJAUPKG6RTPR7F6GTIC6GIA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.136565809 (783h41m50.007648844s) [noise=true , address=145.79.227.29:4946 , piece_id=FR3BLE7ANKJYAQG5GZJ4RL4543FQ55K7VHSJMQGCLC7UDRNKCASA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.137150974 (783h41m50.008234019s) [noise=true , address=145.79.227.35:4487 , piece_id=E5X6GYKRPSYHASZXZB62XA3LXB6EJPTMENDRW6BGEDVDUD5BW4TA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.137656586 (783h41m50.008739621s) [noise=true , address=77.90.31.11:4584 , piece_id=LIDDXUURY5ARSGQ6ETUFION36NJFWMIGJHOSOIE4ZUXJPTJNCGSA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.138273391 (783h41m50.009356436s) [noise=true , address=64.49.13.169:4713 , piece_id=6JKGRVL7KQOZVIIJM3OFQUHWUEF42KMW7Y6I5EPVKVRTKEQVBFNQ, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.138867362 (783h41m50.009950407s) [noise=true , address=64.49.13.155:4916 , piece_id=FPLKLM4RLSO54V7XX3YZF7CGXWSGKNBAKEYQ5FYFYHVE7CTS6PIA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.139440193 (783h41m50.010523238s) [noise=true , address=145.79.227.49:4372 , piece_id=HQ535NWA7OLGPNMYWPMHDJL3ICULP4GEXMYTKLDTC2SPMQYCRZOQ, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.139988797 (783h41m50.011071832s) [noise=true , address=45.91.116.10:4110 , piece_id=VODTQS7BAXPWLTDDTTW75FXYRDBGM2PSJQTIGCRCRNAMMTE55LJA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.140847663 (783h41m50.011930698s) [noise=true , address=45.91.116.14:4200 , piece_id=ON5HNVYRNJTJCNV4JU6NI2ITS2VKZNNMYOBZUZQ7OLXPYAUQL2JQ, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:51.568606354 (783h41m51.439689389s) [upload=685546]: Next returned nil segment
Jan 7 07:10:51.568621312 (783h41m51.439704347s) [upload=685546]: Waiting for error group managing segments...
Jan 7 07:10:51.990916323 (783h41m51.861999368s) [noise=true , address=64.49.13.155:4916 , piece_id=FPLKLM4RLSO54V7XX3YZF7CGXWSGKNBAKEYQ5FYFYHVE7CTS6PIA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:51.994944574 (783h41m51.866027619s) [noise=true , address=145.79.227.44:4227 , piece_id=TCERBF6EX3JHAV63JYLUYA54OHYY33YEM25EQ6CIRS4746KNCULA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.000551040 (783h41m51.871634105s) [noise=true , address=145.79.227.41:4092 , piece_id=3WQEAVSBFU5KF6WVNGACEVQ6M42JEMWW6OHCXPCXVNZCOG4XSMGQ, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.005752454 (783h41m51.876835509s) [noise=true , address=145.79.227.30:4795 , piece_id=G7EGUQIODKUJDE6FEWO5B5C3RQYQ37SAHQYC6BI7X4JS6A7VMOXQ, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.006180148 (783h41m51.877263203s) [noise=true , address=145.79.227.32:4851 , piece_id=5W74ZJOGSKONFPVTJJPHDWNFHLTWRU4KEVX25X4PQCKTRJZR3CZA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.062554132 (783h41m51.933637177s) [noise=true , address=64.49.13.169:4713 , piece_id=6JKGRVL7KQOZVIIJM3OFQUHWUEF42KMW7Y6I5EPVKVRTKEQVBFNQ, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.070238635 (783h41m51.94132167s) [noise=true , address=145.79.227.34:4926 , piece_id=AUBI2PYIJK5YTKFZHL5WF7BQOKNKEPSTZQBERRGEOFZN7RE7T2SA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.088972997 (783h41m51.960056032s) [noise=true , address=145.79.227.33:4897 , piece_id=NLJJNXY2DGHMZWGJRSKGTYDVYZLYZJSVYVIZ35IUCMSI34RD5TBA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.281595960 (783h41m52.152678995s) [noise=true , address=77.90.31.12:4603 , piece_id=HPBI4VUKMT2KFKRW2JTX32HB2YGDGLB2V4BFRVVQ2725Y2IC2LOA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.356241606 (783h41m52.227324641s) [noise=true , address=77.90.31.11:4584 , piece_id=LIDDXUURY5ARSGQ6ETUFION36NJFWMIGJHOSOIE4ZUXJPTJNCGSA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.679275149 (783h41m52.550358204s) [noise=true , address=45.91.116.14:4200 , piece_id=ON5HNVYRNJTJCNV4JU6NI2ITS2VKZNNMYOBZUZQ7OLXPYAUQL2JQ, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.848093704 (783h41m52.719176749s) [noise=true , address=193.149.164.143:4699 , piece_id=DFHC3CPJBT2U7MW5HSNGIGTPMD3QZD4ON62UJ2HNNVWRXOJ4A6PQ, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:53.022732271 (783h41m52.893815316s) [noise=true , address=145.79.227.49:4372 , piece_id=HQ535NWA7OLGPNMYWPMHDJL3ICULP4GEXMYTKLDTC2SPMQYCRZOQ, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:53.048522630 (783h41m52.919605675s) [noise=true , address=145.79.227.36:4543 , piece_id=MUK6THUJD4QHK4VLL3R6DSZP47VV5RJAUPKG6RTPR7F6GTIC6GIA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:53.048547036 (783h41m52.919630082s) [seg_pos={0 0}, upload=685546]: Segment reached optimal threshold of 14 pieces.
Jan 7 07:10:53.048761935 (783h41m52.919844971s) [noise=true , address=77.90.31.16:4741 , piece_id=G4TPC522KLMJBYHYW2SYTUMJFVP4K54R63ZIHOCPQINQO5WE7Q2Q, seg_pos={0 0}, upload=685546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR): protocol: expected piece hash; context canceled; EOF
Jan 7 07:10:53.048771714 (783h41m52.919854749s) [noise=true , address=77.90.31.16:4741 , piece_id=G4TPC522KLMJBYHYW2SYTUMJFVP4K54R63ZIHOCPQINQO5WE7Q2Q, seg_pos={0 0}, upload=685546]: Long tail cancellation due to optimal threshold
Jan 7 07:10:53.049042570 (783h41m52.920125605s) [noise=true , address=145.79.227.29:4946 , piece_id=FR3BLE7ANKJYAQG5GZJ4RL4543FQ55K7VHSJMQGCLC7UDRNKCASA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12v9Q7aigjWPNXyshHmQt9YKrDu5af1vd1TxkDeHeKixSrWgrLX): protocol: expected piece hash; context canceled; EOF
Jan 7 07:10:53.049049483 (783h41m52.920132518s) [noise=true , address=145.79.227.29:4946 , piece_id=FR3BLE7ANKJYAQG5GZJ4RL4543FQ55K7VHSJMQGCLC7UDRNKCASA, seg_pos={0 0}, upload=685546]: Long tail cancellation due to optimal threshold
Jan 7 07:10:53.049146277 (783h41m52.920229322s) [noise=true , address=145.79.227.35:4487 , piece_id=E5X6GYKRPSYHASZXZB62XA3LXB6EJPTMENDRW6BGEDVDUD5BW4TA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AdZLDV7UBbV2DimCHzmoNxycLigvLMx6Fy9U9QB35WTtiV1KX): protocol: expected piece hash; context canceled; EOF
Jan 7 07:10:53.049151166 (783h41m52.920234212s) [noise=true , address=145.79.227.35:4487 , piece_id=E5X6GYKRPSYHASZXZB62XA3LXB6EJPTMENDRW6BGEDVDUD5BW4TA, seg_pos={0 0}, upload=685546]: Long tail cancellation due to optimal threshold
Jan 7 07:10:53.049299398 (783h41m52.920382434s) [noise=true , address=45.91.116.10:4110 , piece_id=VODTQS7BAXPWLTDDTTW75FXYRDBGM2PSJQTIGCRCRNAMMTE55LJA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123KwusATu1ZKtNNVHZZZBE67ZEnVb8puyfQidBi3fEWREqdf1h): protocol: expected piece hash; context canceled; EOF
Jan 7 07:10:53.049308466 (783h41m52.920391512s) [noise=true , address=45.91.116.10:4110 , piece_id=VODTQS7BAXPWLTDDTTW75FXYRDBGM2PSJQTIGCRCRNAMMTE55LJA, seg_pos={0 0}, upload=685546]: Long tail cancellation due to optimal threshold
Jan 7 07:10:53.049324507 (783h41m52.920407542s) [seg_pos={0 0}, upload=685546]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 7 07:10:53.049388278 (783h41m52.920471313s) [upload=685546]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 7 07:10:53.049392767 (783h41m52.920475812s) [upload=685546]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 7 07:10:53.340338951 (783h41m53.211421996s) [upload=685546]: Done uploading segments...
Jan 7 07:10:53.340359300 (783h41m53.211442335s) [upload=685546]: Upload finished. err: <nil>
Duration: 6.537468538s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [.X..XX.X...X....X.........X..X.X..X.......XX.X....X....XX...] duration:
5.753326789s trace:
https://jaeger.dev.storj.io/trace/50ae9c48f3048fc3
FAIL (2026-01-07 07:12:24 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: SOAAPUTTLH3GV2TF2HUAN4BOA7OZKF4QICOHY4S5QEKESZD5N24A) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: BJHBHMTYA5HFEAKMPMM2PSNFBI7X32H2LP6VRY753L32SKGUKCSQ) context deadline exceeded
======LOGS======
Jan 7 07:10:31.248040589 (783h41m31.119123644s) [upload=685543]: Starting upload
Jan 7 07:10:31.248154456 (783h41m31.119237511s) [upload=685543]: Done starting upload
Jan 7 07:10:31.248536263 (783h41m31.119619308s) [upload=685543]: Uploading segments...
Jan 7 07:10:31.248606457 (783h41m31.119689502s) [upload=685543]: Got next segment. Inline: false
Jan 7 07:10:31.248647264 (783h41m31.119730309s) [upload=685543]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 7 07:10:31.248652915 (783h41m31.11973596s) [upload=685543]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 7 07:10:31.492628981 (783h41m31.363712026s) [seg_pos={0 0}, upload=685543]: Begin upload segment...
Jan 7 07:10:31.492693132 (783h41m31.363776177s) [seg_pos={0 0}, upload=685543]: Done begin upload segment.
Jan 7 07:10:31.492797992 (783h41m31.363881027s) [noise=true , address=145.79.227.45:4230 , piece_id=SQFIEICI2DR5H6UQM3D2JITSFG7NCZNBUMDVOUFNXPEAVYYG3UFQ, seg_pos={0 0}, upload=685543]: Uploading piece...
Jan 7 07:10:31.493578138 (783h41m31.364661183s) [noise=true , address=45.91.116.11:4117 , piece_id=Z5XRYSZPRFNSGBJCX66NZADXGM67M5Y7JDPMUWAS3KGFRZ3YTT4Q, seg_pos={0 0}, upload=685543]: Uploading piece...
Jan 7 07:10:31.494131151 (783h41m31.365214186s) [noise=true , address=145.79.227.37:4565 , piece_id=CPOFNSLYHUHQ4JCQMJEX24ZOBPJJCZYZZOAOSEREU4YDCIMLHEOQ, seg_pos={0 0}, upload=685543]: Uploading piece...
Jan 7 07:10:31.494488501 (783h41m31.365571546s) [noise=true , address=77.90.31.19:4841 , piece_id=T4ZU4XYKFBN4Q4FCHIWPFGQKYUBRZFUY2LTNP57EB4RPIR3LCLWQ, seg_pos={0 0}, upload=685543]: Uploading piece...
Jan 7 07:10:31.494776870 (783h41m31.365859915s) [noise=true , address=145.79.227.44:4195 , piece_id=DTUIBJ4JAWQGUOYOVRSMZEHQHBSZHYLISBYXKS22GEHDT7PO2CLA, seg_pos={0 0}, upload=685543]: Uploading piece...
Jan 7 07:10:31.495074447 (783h41m31.366157482s) [noise=true , address=145.79.227.12:4767 , piece_id=SRD2DTDKANNQNF4Q6H6D5QPYJU4SGL42ANBIDDB4AEICFTO6XERQ, seg_pos={0 0}, upload=685543]: Uploading piece...
Jan 7 07:10:31.495142487 (783h41m31.366225532s) [noise=true , address=77.90.31.16:4736 , piece_id=SOAAPUTTLH3GV2TF2HUAN4BOA7OZKF4QICOHY4S5QEKESZD5N24A, seg_pos={0 0}, upload=685543]: Uploading piece...
Jan 7 07:10:31.495649812 (783h41m31.366732847s) [noise=true , address=145.79.227.40:4050 , piece_id=EFGM7CQSPJJRM2ASQWDO6BUDKZ2KQBEGNIHLN4GNQ6SUWKVEEZ3Q, seg_pos={0 0}, upload=685543]: Uploading piece...
Jan 7 07:10:31.496148402 (783h41m31.367231447s) [noise=true , address=145.79.227.31:4799 , piece_id=BCSP53DP7YKGDPF5K37PZJQKD6TN7225D5RBFWOUO7OP4SBHMFBQ, seg_pos={0 0}, upload=685543]: Uploading piece...
Jan 7 07:10:31.496702267 (783h41m31.367785312s) [noise=true , address=77.90.31.11:4554 , piece_id=APWWSHCAFJKZEF5H3FDSJOXMXKKHNJEHO7CRXKYU3B3X4VDMBRZA, seg_pos={0 0}, upload=685543]: Uploading piece...
Jan 7 07:10:31.497256683 (783h41m31.368339728s) [noise=true , address=77.90.31.13:4629 , piece_id=TZJJYNZBZP5CNNLEC6TQHBN4XU4WQKMAHAPFJAUF5H2WTOK2DDCQ, seg_pos={0 0}, upload=685543]: Uploading piece...
Jan 7 07:10:31.499267872 (783h41m31.370350917s) [noise=true , address=145.79.227.26:4654 , piece_id=ODJVMCWVJGMYISGDRP4TLR4ABXXD5ZNUXKOSKDHV2HGVQLNKGD6Q, seg_pos={0 0}, upload=685543]: Uploading piece...
Jan 7 07:10:31.500996664 (783h41m31.372079709s) [noise=true , address=145.79.227.42:4121 , piece_id=RDI73ICBIBVAVGJABHW7TQRSUD6OYAMFZQJM7JLU2N7UT44OCSQQ, seg_pos={0 0}, upload=685543]: Uploading piece...
Jan 7 07:10:31.502637298 (783h41m31.373720333s) [noise=true , address=45.91.116.7:4010 , piece_id=BJHBHMTYA5HFEAKMPMM2PSNFBI7X32H2LP6VRY753L32SKGUKCSQ, seg_pos={0 0}, upload=685543]: Uploading piece...
Jan 7 07:10:31.504268755 (783h41m31.3753518s) [noise=true , address=145.79.227.6:4258 , piece_id=QWU6TB6IVDSS6H5AG4IMHOXPALH2AYZSEM36GTHY5KA6XPKCBXXQ, seg_pos={0 0}, upload=685543]: Uploading piece...
Jan 7 07:10:31.506042783 (783h41m31.377125818s) [noise=true , address=145.79.227.2:4113 , piece_id=YQ54OQ26JK34YPLN3TZXR5LWJOFN3RUVZJ5YUZXGI2RY2OKLRFSA, seg_pos={0 0}, upload=685543]: Uploading piece...
Jan 7 07:10:33.202789827 (783h41m33.073872872s) [upload=685543]: Next returned nil segment
Jan 7 07:10:33.202822870 (783h41m33.073905905s) [upload=685543]: Waiting for error group managing segments...
Jan 7 07:10:33.484280988 (783h41m33.355364023s) [noise=true , address=145.79.227.44:4195 , piece_id=DTUIBJ4JAWQGUOYOVRSMZEHQHBSZHYLISBYXKS22GEHDT7PO2CLA, seg_pos={0 0}, upload=685543]: Done uploading piece. err: <nil>
Jan 7 07:10:33.488021811 (783h41m33.359104846s) [noise=true , address=145.79.227.12:4767 , piece_id=SRD2DTDKANNQNF4Q6H6D5QPYJU4SGL42ANBIDDB4AEICFTO6XERQ, seg_pos={0 0}, upload=685543]: Done uploading piece. err: <nil>
Jan 7 07:10:33.491589184 (783h41m33.362672239s) [noise=true , address=145.79.227.40:4050 , piece_id=EFGM7CQSPJJRM2ASQWDO6BUDKZ2KQBEGNIHLN4GNQ6SUWKVEEZ3Q, seg_pos={0 0}, upload=685543]: Done uploading piece. err: <nil>
Jan 7 07:10:33.498955099 (783h41m33.370038134s) [noise=true , address=145.79.227.6:4258 , piece_id=QWU6TB6IVDSS6H5AG4IMHOXPALH2AYZSEM36GTHY5KA6XPKCBXXQ, seg_pos={0 0}, upload=685543]: Done uploading piece. err: <nil>
Jan 7 07:10:33.501171139 (783h41m33.372254174s) [noise=true , address=145.79.227.37:4565 , piece_id=CPOFNSLYHUHQ4JCQMJEX24ZOBPJJCZYZZOAOSEREU4YDCIMLHEOQ, seg_pos={0 0}, upload=685543]: Done uploading piece. err: <nil>
Jan 7 07:10:33.510816175 (783h41m33.38189922s) [noise=true , address=145.79.227.31:4799 , piece_id=BCSP53DP7YKGDPF5K37PZJQKD6TN7225D5RBFWOUO7OP4SBHMFBQ, seg_pos={0 0}, upload=685543]: Done uploading piece. err: <nil>
Jan 7 07:10:33.511084094 (783h41m33.382167129s) [noise=true , address=145.79.227.45:4230 , piece_id=SQFIEICI2DR5H6UQM3D2JITSFG7NCZNBUMDVOUFNXPEAVYYG3UFQ, seg_pos={0 0}, upload=685543]: Done uploading piece. err: <nil>
Jan 7 07:10:33.516566915 (783h41m33.38764997s) [noise=true , address=145.79.227.26:4654 , piece_id=ODJVMCWVJGMYISGDRP4TLR4ABXXD5ZNUXKOSKDHV2HGVQLNKGD6Q, seg_pos={0 0}, upload=685543]: Done uploading piece. err: <nil>
Jan 7 07:10:33.517421321 (783h41m33.388504357s) [noise=true , address=145.79.227.42:4121 , piece_id=RDI73ICBIBVAVGJABHW7TQRSUD6OYAMFZQJM7JLU2N7UT44OCSQQ, seg_pos={0 0}, upload=685543]: Done uploading piece. err: <nil>
Jan 7 07:10:33.518568827 (783h41m33.389651872s) [noise=true , address=145.79.227.2:4113 , piece_id=YQ54OQ26JK34YPLN3TZXR5LWJOFN3RUVZJ5YUZXGI2RY2OKLRFSA, seg_pos={0 0}, upload=685543]: Done uploading piece. err: <nil>
Jan 7 07:10:33.781941837 (783h41m33.653024882s) [noise=true , address=77.90.31.16:4736 , piece_id=SOAAPUTTLH3GV2TF2HUAN4BOA7OZKF4QICOHY4S5QEKESZD5N24A, seg_pos={0 0}, upload=685543]: Done uploading piece. err: <nil>
Jan 7 07:10:33.851158897 (783h41m33.722241932s) [noise=true , address=77.90.31.13:4629 , piece_id=TZJJYNZBZP5CNNLEC6TQHBN4XU4WQKMAHAPFJAUF5H2WTOK2DDCQ, seg_pos={0 0}, upload=685543]: Done uploading piece. err: <nil>
Jan 7 07:10:33.985266949 (783h41m33.856349984s) [noise=true , address=45.91.116.7:4010 , piece_id=BJHBHMTYA5HFEAKMPMM2PSNFBI7X32H2LP6VRY753L32SKGUKCSQ, seg_pos={0 0}, upload=685543]: Done uploading piece. err: <nil>
Jan 7 07:10:33.985294601 (783h41m33.856377647s) [seg_pos={0 0}, upload=685543]: Segment reached optimal threshold of 13 pieces.
Jan 7 07:10:33.985506545 (783h41m33.85658959s) [noise=true , address=77.90.31.11:4554 , piece_id=APWWSHCAFJKZEF5H3FDSJOXMXKKHNJEHO7CRXKYU3B3X4VDMBRZA, seg_pos={0 0}, upload=685543]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: write tcp 188.166.225.170:43140->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:43140->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43140->77.90.31.11:4554: use of closed network connection
Jan 7 07:10:33.985515993 (783h41m33.856599028s) [noise=true , address=77.90.31.11:4554 , piece_id=APWWSHCAFJKZEF5H3FDSJOXMXKKHNJEHO7CRXKYU3B3X4VDMBRZA, seg_pos={0 0}, upload=685543]: Long tail cancellation due to optimal threshold
Jan 7 07:10:33.985592638 (783h41m33.856675673s) [noise=true , address=77.90.31.19:4841 , piece_id=T4ZU4XYKFBN4Q4FCHIWPFGQKYUBRZFUY2LTNP57EB4RPIR3LCLWQ, seg_pos={0 0}, upload=685543]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF
Jan 7 07:10:33.985599121 (783h41m33.856682156s) [noise=true , address=77.90.31.19:4841 , piece_id=T4ZU4XYKFBN4Q4FCHIWPFGQKYUBRZFUY2LTNP57EB4RPIR3LCLWQ, seg_pos={0 0}, upload=685543]: Long tail cancellation due to optimal threshold
Jan 7 07:10:33.985754036 (783h41m33.856837071s) [noise=true , address=45.91.116.11:4117 , piece_id=Z5XRYSZPRFNSGBJCX66NZADXGM67M5Y7JDPMUWAS3KGFRZ3YTT4Q, seg_pos={0 0}, upload=685543]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF
Jan 7 07:10:33.985759727 (783h41m33.856842762s) [noise=true , address=45.91.116.11:4117 , piece_id=Z5XRYSZPRFNSGBJCX66NZADXGM67M5Y7JDPMUWAS3KGFRZ3YTT4Q, seg_pos={0 0}, upload=685543]: Long tail cancellation due to optimal threshold
Jan 7 07:10:33.985872070 (783h41m33.856955106s) [seg_pos={0 0}, upload=685543]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 7 07:10:33.985962032 (783h41m33.857045077s) [upload=685543]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 7 07:10:33.985966591 (783h41m33.857049636s) [upload=685543]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 7 07:10:34.275584985 (783h41m34.14666802s) [upload=685543]: Done uploading segments...
Jan 7 07:10:34.275603481 (783h41m34.146686516s) [upload=685543]: Upload finished. err: <nil>
Duration: 5.753354632s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [X....X..............X...............X...................X...] duration:
15.353915107s trace:
https://jaeger.dev.storj.io/trace/36161e151ba11c34
FAIL (2026-01-07 07:11:57 UTC)
orbiter warning: context deadline exceeded
Duration: 15.353939634s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [.......X........................................X........X..] duration:
15.187832625s trace:
https://jaeger.dev.storj.io/trace/1ea01aa868761fdc
FAIL (2026-01-07 07:11:54 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
133194519 (783h41m50.004277565s) [noise=true , address=145.79.227.32:4851 , piece_id=5W74ZJOGSKONFPVTJJPHDWNFHLTWRU4KEVX25X4PQCKTRJZR3CZA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.133698059 (783h41m50.004781104s) [noise=true , address=145.79.227.33:4897 , piece_id=NLJJNXY2DGHMZWGJRSKGTYDVYZLYZJSVYVIZ35IUCMSI34RD5TBA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.134246032 (783h41m50.005329067s) [noise=true , address=145.79.227.44:4227 , piece_id=TCERBF6EX3JHAV63JYLUYA54OHYY33YEM25EQ6CIRS4746KNCULA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.134617359 (783h41m50.005700404s) [noise=true , address=145.79.227.30:4795 , piece_id=G7EGUQIODKUJDE6FEWO5B5C3RQYQ37SAHQYC6BI7X4JS6A7VMOXQ, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.135018602 (783h41m50.006101637s) [noise=true , address=77.90.31.16:4741 , piece_id=G4TPC522KLMJBYHYW2SYTUMJFVP4K54R63ZIHOCPQINQO5WE7Q2Q, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.135492385 (783h41m50.00657542s) [noise=true , address=145.79.227.41:4092 , piece_id=3WQEAVSBFU5KF6WVNGACEVQ6M42JEMWW6OHCXPCXVNZCOG4XSMGQ, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.135895753 (783h41m50.006978799s) [noise=true , address=145.79.227.34:4926 , piece_id=AUBI2PYIJK5YTKFZHL5WF7BQOKNKEPSTZQBERRGEOFZN7RE7T2SA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.136238677 (783h41m50.007321722s) [noise=true , address=193.149.164.143:4699 , piece_id=DFHC3CPJBT2U7MW5HSNGIGTPMD3QZD4ON62UJ2HNNVWRXOJ4A6PQ, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.136546201 (783h41m50.007629236s) [noise=true , address=145.79.227.36:4543 , piece_id=MUK6THUJD4QHK4VLL3R6DSZP47VV5RJAUPKG6RTPR7F6GTIC6GIA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.136565809 (783h41m50.007648844s) [noise=true , address=145.79.227.29:4946 , piece_id=FR3BLE7ANKJYAQG5GZJ4RL4543FQ55K7VHSJMQGCLC7UDRNKCASA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.137150974 (783h41m50.008234019s) [noise=true , address=145.79.227.35:4487 , piece_id=E5X6GYKRPSYHASZXZB62XA3LXB6EJPTMENDRW6BGEDVDUD5BW4TA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.137656586 (783h41m50.008739621s) [noise=true , address=77.90.31.11:4584 , piece_id=LIDDXUURY5ARSGQ6ETUFION36NJFWMIGJHOSOIE4ZUXJPTJNCGSA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.138273391 (783h41m50.009356436s) [noise=true , address=64.49.13.169:4713 , piece_id=6JKGRVL7KQOZVIIJM3OFQUHWUEF42KMW7Y6I5EPVKVRTKEQVBFNQ, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.138867362 (783h41m50.009950407s) [noise=true , address=64.49.13.155:4916 , piece_id=FPLKLM4RLSO54V7XX3YZF7CGXWSGKNBAKEYQ5FYFYHVE7CTS6PIA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.139440193 (783h41m50.010523238s) [noise=true , address=145.79.227.49:4372 , piece_id=HQ535NWA7OLGPNMYWPMHDJL3ICULP4GEXMYTKLDTC2SPMQYCRZOQ, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.139988797 (783h41m50.011071832s) [noise=true , address=45.91.116.10:4110 , piece_id=VODTQS7BAXPWLTDDTTW75FXYRDBGM2PSJQTIGCRCRNAMMTE55LJA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.140847663 (783h41m50.011930698s) [noise=true , address=45.91.116.14:4200 , piece_id=ON5HNVYRNJTJCNV4JU6NI2ITS2VKZNNMYOBZUZQ7OLXPYAUQL2JQ, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:51.568606354 (783h41m51.439689389s) [upload=685546]: Next returned nil segment
Jan 7 07:10:51.568621312 (783h41m51.439704347s) [upload=685546]: Waiting for error group managing segments...
Jan 7 07:10:51.990916323 (783h41m51.861999368s) [noise=true , address=64.49.13.155:4916 , piece_id=FPLKLM4RLSO54V7XX3YZF7CGXWSGKNBAKEYQ5FYFYHVE7CTS6PIA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:51.994944574 (783h41m51.866027619s) [noise=true , address=145.79.227.44:4227 , piece_id=TCERBF6EX3JHAV63JYLUYA54OHYY33YEM25EQ6CIRS4746KNCULA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.000551040 (783h41m51.871634105s) [noise=true , address=145.79.227.41:4092 , piece_id=3WQEAVSBFU5KF6WVNGACEVQ6M42JEMWW6OHCXPCXVNZCOG4XSMGQ, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.005752454 (783h41m51.876835509s) [noise=true , address=145.79.227.30:4795 , piece_id=G7EGUQIODKUJDE6FEWO5B5C3RQYQ37SAHQYC6BI7X4JS6A7VMOXQ, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.006180148 (783h41m51.877263203s) [noise=true , address=145.79.227.32:4851 , piece_id=5W74ZJOGSKONFPVTJJPHDWNFHLTWRU4KEVX25X4PQCKTRJZR3CZA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.062554132 (783h41m51.933637177s) [noise=true , address=64.49.13.169:4713 , piece_id=6JKGRVL7KQOZVIIJM3OFQUHWUEF42KMW7Y6I5EPVKVRTKEQVBFNQ, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.070238635 (783h41m51.94132167s) [noise=true , address=145.79.227.34:4926 , piece_id=AUBI2PYIJK5YTKFZHL5WF7BQOKNKEPSTZQBERRGEOFZN7RE7T2SA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.088972997 (783h41m51.960056032s) [noise=true , address=145.79.227.33:4897 , piece_id=NLJJNXY2DGHMZWGJRSKGTYDVYZLYZJSVYVIZ35IUCMSI34RD5TBA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.281595960 (783h41m52.152678995s) [noise=true , address=77.90.31.12:4603 , piece_id=HPBI4VUKMT2KFKRW2JTX32HB2YGDGLB2V4BFRVVQ2725Y2IC2LOA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.356241606 (783h41m52.227324641s) [noise=true , address=77.90.31.11:4584 , piece_id=LIDDXUURY5ARSGQ6ETUFION36NJFWMIGJHOSOIE4ZUXJPTJNCGSA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.679275149 (783h41m52.550358204s) [noise=true , address=45.91.116.14:4200 , piece_id=ON5HNVYRNJTJCNV4JU6NI2ITS2VKZNNMYOBZUZQ7OLXPYAUQL2JQ, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.848093704 (783h41m52.719176749s) [noise=true , address=193.149.164.143:4699 , piece_id=DFHC3CPJBT2U7MW5HSNGIGTPMD3QZD4ON62UJ2HNNVWRXOJ4A6PQ, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:53.022732271 (783h41m52.893815316s) [noise=true , address=145.79.227.49:4372 , piece_id=HQ535NWA7OLGPNMYWPMHDJL3ICULP4GEXMYTKLDTC2SPMQYCRZOQ, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:53.048522630 (783h41m52.919605675s) [noise=true , address=145.79.227.36:4543 , piece_id=MUK6THUJD4QHK4VLL3R6DSZP47VV5RJAUPKG6RTPR7F6GTIC6GIA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:53.048547036 (783h41m52.919630082s) [seg_pos={0 0}, upload=685546]: Segment reached optimal threshold of 14 pieces.
Jan 7 07:10:53.048761935 (783h41m52.919844971s) [noise=true , address=77.90.31.16:4741 , piece_id=G4TPC522KLMJBYHYW2SYTUMJFVP4K54R63ZIHOCPQINQO5WE7Q2Q, seg_pos={0 0}, upload=685546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR): protocol: expected piece hash; context canceled; EOF
Jan 7 07:10:53.048771714 (783h41m52.919854749s) [noise=true , address=77.90.31.16:4741 , piece_id=G4TPC522KLMJBYHYW2SYTUMJFVP4K54R63ZIHOCPQINQO5WE7Q2Q, seg_pos={0 0}, upload=685546]: Long tail cancellation due to optimal threshold
Jan 7 07:10:53.049042570 (783h41m52.920125605s) [noise=true , address=145.79.227.29:4946 , piece_id=FR3BLE7ANKJYAQG5GZJ4RL4543FQ55K7VHSJMQGCLC7UDRNKCASA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12v9Q7aigjWPNXyshHmQt9YKrDu5af1vd1TxkDeHeKixSrWgrLX): protocol: expected piece hash; context canceled; EOF
Jan 7 07:10:53.049049483 (783h41m52.920132518s) [noise=true , address=145.79.227.29:4946 , piece_id=FR3BLE7ANKJYAQG5GZJ4RL4543FQ55K7VHSJMQGCLC7UDRNKCASA, seg_pos={0 0}, upload=685546]: Long tail cancellation due to optimal threshold
Jan 7 07:10:53.049146277 (783h41m52.920229322s) [noise=true , address=145.79.227.35:4487 , piece_id=E5X6GYKRPSYHASZXZB62XA3LXB6EJPTMENDRW6BGEDVDUD5BW4TA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AdZLDV7UBbV2DimCHzmoNxycLigvLMx6Fy9U9QB35WTtiV1KX): protocol: expected piece hash; context canceled; EOF
Jan 7 07:10:53.049151166 (783h41m52.920234212s) [noise=true , address=145.79.227.35:4487 , piece_id=E5X6GYKRPSYHASZXZB62XA3LXB6EJPTMENDRW6BGEDVDUD5BW4TA, seg_pos={0 0}, upload=685546]: Long tail cancellation due to optimal threshold
Jan 7 07:10:53.049299398 (783h41m52.920382434s) [noise=true , address=45.91.116.10:4110 , piece_id=VODTQS7BAXPWLTDDTTW75FXYRDBGM2PSJQTIGCRCRNAMMTE55LJA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123KwusATu1ZKtNNVHZZZBE67ZEnVb8puyfQidBi3fEWREqdf1h): protocol: expected piece hash; context canceled; EOF
Jan 7 07:10:53.049308466 (783h41m52.920391512s) [noise=true , address=45.91.116.10:4110 , piece_id=VODTQS7BAXPWLTDDTTW75FXYRDBGM2PSJQTIGCRCRNAMMTE55LJA, seg_pos={0 0}, upload=685546]: Long tail cancellation due to optimal threshold
Jan 7 07:10:53.049324507 (783h41m52.920407542s) [seg_pos={0 0}, upload=685546]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 7 07:10:53.049388278 (783h41m52.920471313s) [upload=685546]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 7 07:10:53.049392767 (783h41m52.920475812s) [upload=685546]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 7 07:10:53.340338951 (783h41m53.211421996s) [upload=685546]: Done uploading segments...
Jan 7 07:10:53.340359300 (783h41m53.211442335s) [upload=685546]: Upload finished. err: <nil>
Duration: 15.187881519s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [..........X.XXX.X.X..X....X.X.....X....X.X.....X.......X....] duration:
24.181462509s trace:
https://jaeger.dev.storj.io/trace/ae99c37afb2c1b6
FAIL (2026-01-07 07:11:50 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 24.181487167s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
49 out of 60 is failed (threshold=6) [X.XXXXX.XXXXXX.XXXXXXXXXXX.XXXX.XXXX..XXXXX.XXXXX..XXXXX.XXX] duration:
5.001198734s trace:
https://jaeger.dev.storj.io/trace/4736fb73cd9c60d8
FAIL (2026-01-07 07:11:40 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001228781s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [........X........................................X........X.] duration:
5.930632159s trace:
https://jaeger.dev.storj.io/trace/1ea01aa868761fdc
FAIL (2026-01-07 07:11:29 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
133194519 (783h41m50.004277565s) [noise=true , address=145.79.227.32:4851 , piece_id=5W74ZJOGSKONFPVTJJPHDWNFHLTWRU4KEVX25X4PQCKTRJZR3CZA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.133698059 (783h41m50.004781104s) [noise=true , address=145.79.227.33:4897 , piece_id=NLJJNXY2DGHMZWGJRSKGTYDVYZLYZJSVYVIZ35IUCMSI34RD5TBA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.134246032 (783h41m50.005329067s) [noise=true , address=145.79.227.44:4227 , piece_id=TCERBF6EX3JHAV63JYLUYA54OHYY33YEM25EQ6CIRS4746KNCULA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.134617359 (783h41m50.005700404s) [noise=true , address=145.79.227.30:4795 , piece_id=G7EGUQIODKUJDE6FEWO5B5C3RQYQ37SAHQYC6BI7X4JS6A7VMOXQ, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.135018602 (783h41m50.006101637s) [noise=true , address=77.90.31.16:4741 , piece_id=G4TPC522KLMJBYHYW2SYTUMJFVP4K54R63ZIHOCPQINQO5WE7Q2Q, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.135492385 (783h41m50.00657542s) [noise=true , address=145.79.227.41:4092 , piece_id=3WQEAVSBFU5KF6WVNGACEVQ6M42JEMWW6OHCXPCXVNZCOG4XSMGQ, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.135895753 (783h41m50.006978799s) [noise=true , address=145.79.227.34:4926 , piece_id=AUBI2PYIJK5YTKFZHL5WF7BQOKNKEPSTZQBERRGEOFZN7RE7T2SA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.136238677 (783h41m50.007321722s) [noise=true , address=193.149.164.143:4699 , piece_id=DFHC3CPJBT2U7MW5HSNGIGTPMD3QZD4ON62UJ2HNNVWRXOJ4A6PQ, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.136546201 (783h41m50.007629236s) [noise=true , address=145.79.227.36:4543 , piece_id=MUK6THUJD4QHK4VLL3R6DSZP47VV5RJAUPKG6RTPR7F6GTIC6GIA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.136565809 (783h41m50.007648844s) [noise=true , address=145.79.227.29:4946 , piece_id=FR3BLE7ANKJYAQG5GZJ4RL4543FQ55K7VHSJMQGCLC7UDRNKCASA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.137150974 (783h41m50.008234019s) [noise=true , address=145.79.227.35:4487 , piece_id=E5X6GYKRPSYHASZXZB62XA3LXB6EJPTMENDRW6BGEDVDUD5BW4TA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.137656586 (783h41m50.008739621s) [noise=true , address=77.90.31.11:4584 , piece_id=LIDDXUURY5ARSGQ6ETUFION36NJFWMIGJHOSOIE4ZUXJPTJNCGSA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.138273391 (783h41m50.009356436s) [noise=true , address=64.49.13.169:4713 , piece_id=6JKGRVL7KQOZVIIJM3OFQUHWUEF42KMW7Y6I5EPVKVRTKEQVBFNQ, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.138867362 (783h41m50.009950407s) [noise=true , address=64.49.13.155:4916 , piece_id=FPLKLM4RLSO54V7XX3YZF7CGXWSGKNBAKEYQ5FYFYHVE7CTS6PIA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.139440193 (783h41m50.010523238s) [noise=true , address=145.79.227.49:4372 , piece_id=HQ535NWA7OLGPNMYWPMHDJL3ICULP4GEXMYTKLDTC2SPMQYCRZOQ, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.139988797 (783h41m50.011071832s) [noise=true , address=45.91.116.10:4110 , piece_id=VODTQS7BAXPWLTDDTTW75FXYRDBGM2PSJQTIGCRCRNAMMTE55LJA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.140847663 (783h41m50.011930698s) [noise=true , address=45.91.116.14:4200 , piece_id=ON5HNVYRNJTJCNV4JU6NI2ITS2VKZNNMYOBZUZQ7OLXPYAUQL2JQ, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:51.568606354 (783h41m51.439689389s) [upload=685546]: Next returned nil segment
Jan 7 07:10:51.568621312 (783h41m51.439704347s) [upload=685546]: Waiting for error group managing segments...
Jan 7 07:10:51.990916323 (783h41m51.861999368s) [noise=true , address=64.49.13.155:4916 , piece_id=FPLKLM4RLSO54V7XX3YZF7CGXWSGKNBAKEYQ5FYFYHVE7CTS6PIA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:51.994944574 (783h41m51.866027619s) [noise=true , address=145.79.227.44:4227 , piece_id=TCERBF6EX3JHAV63JYLUYA54OHYY33YEM25EQ6CIRS4746KNCULA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.000551040 (783h41m51.871634105s) [noise=true , address=145.79.227.41:4092 , piece_id=3WQEAVSBFU5KF6WVNGACEVQ6M42JEMWW6OHCXPCXVNZCOG4XSMGQ, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.005752454 (783h41m51.876835509s) [noise=true , address=145.79.227.30:4795 , piece_id=G7EGUQIODKUJDE6FEWO5B5C3RQYQ37SAHQYC6BI7X4JS6A7VMOXQ, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.006180148 (783h41m51.877263203s) [noise=true , address=145.79.227.32:4851 , piece_id=5W74ZJOGSKONFPVTJJPHDWNFHLTWRU4KEVX25X4PQCKTRJZR3CZA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.062554132 (783h41m51.933637177s) [noise=true , address=64.49.13.169:4713 , piece_id=6JKGRVL7KQOZVIIJM3OFQUHWUEF42KMW7Y6I5EPVKVRTKEQVBFNQ, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.070238635 (783h41m51.94132167s) [noise=true , address=145.79.227.34:4926 , piece_id=AUBI2PYIJK5YTKFZHL5WF7BQOKNKEPSTZQBERRGEOFZN7RE7T2SA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.088972997 (783h41m51.960056032s) [noise=true , address=145.79.227.33:4897 , piece_id=NLJJNXY2DGHMZWGJRSKGTYDVYZLYZJSVYVIZ35IUCMSI34RD5TBA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.281595960 (783h41m52.152678995s) [noise=true , address=77.90.31.12:4603 , piece_id=HPBI4VUKMT2KFKRW2JTX32HB2YGDGLB2V4BFRVVQ2725Y2IC2LOA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.356241606 (783h41m52.227324641s) [noise=true , address=77.90.31.11:4584 , piece_id=LIDDXUURY5ARSGQ6ETUFION36NJFWMIGJHOSOIE4ZUXJPTJNCGSA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.679275149 (783h41m52.550358204s) [noise=true , address=45.91.116.14:4200 , piece_id=ON5HNVYRNJTJCNV4JU6NI2ITS2VKZNNMYOBZUZQ7OLXPYAUQL2JQ, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.848093704 (783h41m52.719176749s) [noise=true , address=193.149.164.143:4699 , piece_id=DFHC3CPJBT2U7MW5HSNGIGTPMD3QZD4ON62UJ2HNNVWRXOJ4A6PQ, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:53.022732271 (783h41m52.893815316s) [noise=true , address=145.79.227.49:4372 , piece_id=HQ535NWA7OLGPNMYWPMHDJL3ICULP4GEXMYTKLDTC2SPMQYCRZOQ, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:53.048522630 (783h41m52.919605675s) [noise=true , address=145.79.227.36:4543 , piece_id=MUK6THUJD4QHK4VLL3R6DSZP47VV5RJAUPKG6RTPR7F6GTIC6GIA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:53.048547036 (783h41m52.919630082s) [seg_pos={0 0}, upload=685546]: Segment reached optimal threshold of 14 pieces.
Jan 7 07:10:53.048761935 (783h41m52.919844971s) [noise=true , address=77.90.31.16:4741 , piece_id=G4TPC522KLMJBYHYW2SYTUMJFVP4K54R63ZIHOCPQINQO5WE7Q2Q, seg_pos={0 0}, upload=685546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR): protocol: expected piece hash; context canceled; EOF
Jan 7 07:10:53.048771714 (783h41m52.919854749s) [noise=true , address=77.90.31.16:4741 , piece_id=G4TPC522KLMJBYHYW2SYTUMJFVP4K54R63ZIHOCPQINQO5WE7Q2Q, seg_pos={0 0}, upload=685546]: Long tail cancellation due to optimal threshold
Jan 7 07:10:53.049042570 (783h41m52.920125605s) [noise=true , address=145.79.227.29:4946 , piece_id=FR3BLE7ANKJYAQG5GZJ4RL4543FQ55K7VHSJMQGCLC7UDRNKCASA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12v9Q7aigjWPNXyshHmQt9YKrDu5af1vd1TxkDeHeKixSrWgrLX): protocol: expected piece hash; context canceled; EOF
Jan 7 07:10:53.049049483 (783h41m52.920132518s) [noise=true , address=145.79.227.29:4946 , piece_id=FR3BLE7ANKJYAQG5GZJ4RL4543FQ55K7VHSJMQGCLC7UDRNKCASA, seg_pos={0 0}, upload=685546]: Long tail cancellation due to optimal threshold
Jan 7 07:10:53.049146277 (783h41m52.920229322s) [noise=true , address=145.79.227.35:4487 , piece_id=E5X6GYKRPSYHASZXZB62XA3LXB6EJPTMENDRW6BGEDVDUD5BW4TA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AdZLDV7UBbV2DimCHzmoNxycLigvLMx6Fy9U9QB35WTtiV1KX): protocol: expected piece hash; context canceled; EOF
Jan 7 07:10:53.049151166 (783h41m52.920234212s) [noise=true , address=145.79.227.35:4487 , piece_id=E5X6GYKRPSYHASZXZB62XA3LXB6EJPTMENDRW6BGEDVDUD5BW4TA, seg_pos={0 0}, upload=685546]: Long tail cancellation due to optimal threshold
Jan 7 07:10:53.049299398 (783h41m52.920382434s) [noise=true , address=45.91.116.10:4110 , piece_id=VODTQS7BAXPWLTDDTTW75FXYRDBGM2PSJQTIGCRCRNAMMTE55LJA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123KwusATu1ZKtNNVHZZZBE67ZEnVb8puyfQidBi3fEWREqdf1h): protocol: expected piece hash; context canceled; EOF
Jan 7 07:10:53.049308466 (783h41m52.920391512s) [noise=true , address=45.91.116.10:4110 , piece_id=VODTQS7BAXPWLTDDTTW75FXYRDBGM2PSJQTIGCRCRNAMMTE55LJA, seg_pos={0 0}, upload=685546]: Long tail cancellation due to optimal threshold
Jan 7 07:10:53.049324507 (783h41m52.920407542s) [seg_pos={0 0}, upload=685546]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 7 07:10:53.049388278 (783h41m52.920471313s) [upload=685546]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 7 07:10:53.049392767 (783h41m52.920475812s) [upload=685546]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 7 07:10:53.340338951 (783h41m53.211421996s) [upload=685546]: Done uploading segments...
Jan 7 07:10:53.340359300 (783h41m53.211442335s) [upload=685546]: Upload finished. err: <nil>
Duration: 5.930648421s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [..X..XX.X...X....X.........X..X.X..X.......XX.X....X....XX..] duration:
6.651004576s trace:
https://jaeger.dev.storj.io/trace/50ae9c48f3048fc3
FAIL (2026-01-07 07:11:24 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: SOAAPUTTLH3GV2TF2HUAN4BOA7OZKF4QICOHY4S5QEKESZD5N24A) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: BJHBHMTYA5HFEAKMPMM2PSNFBI7X32H2LP6VRY753L32SKGUKCSQ) context deadline exceeded
======LOGS======
Jan 7 07:10:31.248040589 (783h41m31.119123644s) [upload=685543]: Starting upload
Jan 7 07:10:31.248154456 (783h41m31.119237511s) [upload=685543]: Done starting upload
Jan 7 07:10:31.248536263 (783h41m31.119619308s) [upload=685543]: Uploading segments...
Jan 7 07:10:31.248606457 (783h41m31.119689502s) [upload=685543]: Got next segment. Inline: false
Jan 7 07:10:31.248647264 (783h41m31.119730309s) [upload=685543]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 7 07:10:31.248652915 (783h41m31.11973596s) [upload=685543]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 7 07:10:31.492628981 (783h41m31.363712026s) [seg_pos={0 0}, upload=685543]: Begin upload segment...
Jan 7 07:10:31.492693132 (783h41m31.363776177s) [seg_pos={0 0}, upload=685543]: Done begin upload segment.
Jan 7 07:10:31.492797992 (783h41m31.363881027s) [noise=true , address=145.79.227.45:4230 , piece_id=SQFIEICI2DR5H6UQM3D2JITSFG7NCZNBUMDVOUFNXPEAVYYG3UFQ, seg_pos={0 0}, upload=685543]: Uploading piece...
Jan 7 07:10:31.493578138 (783h41m31.364661183s) [noise=true , address=45.91.116.11:4117 , piece_id=Z5XRYSZPRFNSGBJCX66NZADXGM67M5Y7JDPMUWAS3KGFRZ3YTT4Q, seg_pos={0 0}, upload=685543]: Uploading piece...
Jan 7 07:10:31.494131151 (783h41m31.365214186s) [noise=true , address=145.79.227.37:4565 , piece_id=CPOFNSLYHUHQ4JCQMJEX24ZOBPJJCZYZZOAOSEREU4YDCIMLHEOQ, seg_pos={0 0}, upload=685543]: Uploading piece...
Jan 7 07:10:31.494488501 (783h41m31.365571546s) [noise=true , address=77.90.31.19:4841 , piece_id=T4ZU4XYKFBN4Q4FCHIWPFGQKYUBRZFUY2LTNP57EB4RPIR3LCLWQ, seg_pos={0 0}, upload=685543]: Uploading piece...
Jan 7 07:10:31.494776870 (783h41m31.365859915s) [noise=true , address=145.79.227.44:4195 , piece_id=DTUIBJ4JAWQGUOYOVRSMZEHQHBSZHYLISBYXKS22GEHDT7PO2CLA, seg_pos={0 0}, upload=685543]: Uploading piece...
Jan 7 07:10:31.495074447 (783h41m31.366157482s) [noise=true , address=145.79.227.12:4767 , piece_id=SRD2DTDKANNQNF4Q6H6D5QPYJU4SGL42ANBIDDB4AEICFTO6XERQ, seg_pos={0 0}, upload=685543]: Uploading piece...
Jan 7 07:10:31.495142487 (783h41m31.366225532s) [noise=true , address=77.90.31.16:4736 , piece_id=SOAAPUTTLH3GV2TF2HUAN4BOA7OZKF4QICOHY4S5QEKESZD5N24A, seg_pos={0 0}, upload=685543]: Uploading piece...
Jan 7 07:10:31.495649812 (783h41m31.366732847s) [noise=true , address=145.79.227.40:4050 , piece_id=EFGM7CQSPJJRM2ASQWDO6BUDKZ2KQBEGNIHLN4GNQ6SUWKVEEZ3Q, seg_pos={0 0}, upload=685543]: Uploading piece...
Jan 7 07:10:31.496148402 (783h41m31.367231447s) [noise=true , address=145.79.227.31:4799 , piece_id=BCSP53DP7YKGDPF5K37PZJQKD6TN7225D5RBFWOUO7OP4SBHMFBQ, seg_pos={0 0}, upload=685543]: Uploading piece...
Jan 7 07:10:31.496702267 (783h41m31.367785312s) [noise=true , address=77.90.31.11:4554 , piece_id=APWWSHCAFJKZEF5H3FDSJOXMXKKHNJEHO7CRXKYU3B3X4VDMBRZA, seg_pos={0 0}, upload=685543]: Uploading piece...
Jan 7 07:10:31.497256683 (783h41m31.368339728s) [noise=true , address=77.90.31.13:4629 , piece_id=TZJJYNZBZP5CNNLEC6TQHBN4XU4WQKMAHAPFJAUF5H2WTOK2DDCQ, seg_pos={0 0}, upload=685543]: Uploading piece...
Jan 7 07:10:31.499267872 (783h41m31.370350917s) [noise=true , address=145.79.227.26:4654 , piece_id=ODJVMCWVJGMYISGDRP4TLR4ABXXD5ZNUXKOSKDHV2HGVQLNKGD6Q, seg_pos={0 0}, upload=685543]: Uploading piece...
Jan 7 07:10:31.500996664 (783h41m31.372079709s) [noise=true , address=145.79.227.42:4121 , piece_id=RDI73ICBIBVAVGJABHW7TQRSUD6OYAMFZQJM7JLU2N7UT44OCSQQ, seg_pos={0 0}, upload=685543]: Uploading piece...
Jan 7 07:10:31.502637298 (783h41m31.373720333s) [noise=true , address=45.91.116.7:4010 , piece_id=BJHBHMTYA5HFEAKMPMM2PSNFBI7X32H2LP6VRY753L32SKGUKCSQ, seg_pos={0 0}, upload=685543]: Uploading piece...
Jan 7 07:10:31.504268755 (783h41m31.3753518s) [noise=true , address=145.79.227.6:4258 , piece_id=QWU6TB6IVDSS6H5AG4IMHOXPALH2AYZSEM36GTHY5KA6XPKCBXXQ, seg_pos={0 0}, upload=685543]: Uploading piece...
Jan 7 07:10:31.506042783 (783h41m31.377125818s) [noise=true , address=145.79.227.2:4113 , piece_id=YQ54OQ26JK34YPLN3TZXR5LWJOFN3RUVZJ5YUZXGI2RY2OKLRFSA, seg_pos={0 0}, upload=685543]: Uploading piece...
Jan 7 07:10:33.202789827 (783h41m33.073872872s) [upload=685543]: Next returned nil segment
Jan 7 07:10:33.202822870 (783h41m33.073905905s) [upload=685543]: Waiting for error group managing segments...
Jan 7 07:10:33.484280988 (783h41m33.355364023s) [noise=true , address=145.79.227.44:4195 , piece_id=DTUIBJ4JAWQGUOYOVRSMZEHQHBSZHYLISBYXKS22GEHDT7PO2CLA, seg_pos={0 0}, upload=685543]: Done uploading piece. err: <nil>
Jan 7 07:10:33.488021811 (783h41m33.359104846s) [noise=true , address=145.79.227.12:4767 , piece_id=SRD2DTDKANNQNF4Q6H6D5QPYJU4SGL42ANBIDDB4AEICFTO6XERQ, seg_pos={0 0}, upload=685543]: Done uploading piece. err: <nil>
Jan 7 07:10:33.491589184 (783h41m33.362672239s) [noise=true , address=145.79.227.40:4050 , piece_id=EFGM7CQSPJJRM2ASQWDO6BUDKZ2KQBEGNIHLN4GNQ6SUWKVEEZ3Q, seg_pos={0 0}, upload=685543]: Done uploading piece. err: <nil>
Jan 7 07:10:33.498955099 (783h41m33.370038134s) [noise=true , address=145.79.227.6:4258 , piece_id=QWU6TB6IVDSS6H5AG4IMHOXPALH2AYZSEM36GTHY5KA6XPKCBXXQ, seg_pos={0 0}, upload=685543]: Done uploading piece. err: <nil>
Jan 7 07:10:33.501171139 (783h41m33.372254174s) [noise=true , address=145.79.227.37:4565 , piece_id=CPOFNSLYHUHQ4JCQMJEX24ZOBPJJCZYZZOAOSEREU4YDCIMLHEOQ, seg_pos={0 0}, upload=685543]: Done uploading piece. err: <nil>
Jan 7 07:10:33.510816175 (783h41m33.38189922s) [noise=true , address=145.79.227.31:4799 , piece_id=BCSP53DP7YKGDPF5K37PZJQKD6TN7225D5RBFWOUO7OP4SBHMFBQ, seg_pos={0 0}, upload=685543]: Done uploading piece. err: <nil>
Jan 7 07:10:33.511084094 (783h41m33.382167129s) [noise=true , address=145.79.227.45:4230 , piece_id=SQFIEICI2DR5H6UQM3D2JITSFG7NCZNBUMDVOUFNXPEAVYYG3UFQ, seg_pos={0 0}, upload=685543]: Done uploading piece. err: <nil>
Jan 7 07:10:33.516566915 (783h41m33.38764997s) [noise=true , address=145.79.227.26:4654 , piece_id=ODJVMCWVJGMYISGDRP4TLR4ABXXD5ZNUXKOSKDHV2HGVQLNKGD6Q, seg_pos={0 0}, upload=685543]: Done uploading piece. err: <nil>
Jan 7 07:10:33.517421321 (783h41m33.388504357s) [noise=true , address=145.79.227.42:4121 , piece_id=RDI73ICBIBVAVGJABHW7TQRSUD6OYAMFZQJM7JLU2N7UT44OCSQQ, seg_pos={0 0}, upload=685543]: Done uploading piece. err: <nil>
Jan 7 07:10:33.518568827 (783h41m33.389651872s) [noise=true , address=145.79.227.2:4113 , piece_id=YQ54OQ26JK34YPLN3TZXR5LWJOFN3RUVZJ5YUZXGI2RY2OKLRFSA, seg_pos={0 0}, upload=685543]: Done uploading piece. err: <nil>
Jan 7 07:10:33.781941837 (783h41m33.653024882s) [noise=true , address=77.90.31.16:4736 , piece_id=SOAAPUTTLH3GV2TF2HUAN4BOA7OZKF4QICOHY4S5QEKESZD5N24A, seg_pos={0 0}, upload=685543]: Done uploading piece. err: <nil>
Jan 7 07:10:33.851158897 (783h41m33.722241932s) [noise=true , address=77.90.31.13:4629 , piece_id=TZJJYNZBZP5CNNLEC6TQHBN4XU4WQKMAHAPFJAUF5H2WTOK2DDCQ, seg_pos={0 0}, upload=685543]: Done uploading piece. err: <nil>
Jan 7 07:10:33.985266949 (783h41m33.856349984s) [noise=true , address=45.91.116.7:4010 , piece_id=BJHBHMTYA5HFEAKMPMM2PSNFBI7X32H2LP6VRY753L32SKGUKCSQ, seg_pos={0 0}, upload=685543]: Done uploading piece. err: <nil>
Jan 7 07:10:33.985294601 (783h41m33.856377647s) [seg_pos={0 0}, upload=685543]: Segment reached optimal threshold of 13 pieces.
Jan 7 07:10:33.985506545 (783h41m33.85658959s) [noise=true , address=77.90.31.11:4554 , piece_id=APWWSHCAFJKZEF5H3FDSJOXMXKKHNJEHO7CRXKYU3B3X4VDMBRZA, seg_pos={0 0}, upload=685543]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: write tcp 188.166.225.170:43140->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:43140->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43140->77.90.31.11:4554: use of closed network connection
Jan 7 07:10:33.985515993 (783h41m33.856599028s) [noise=true , address=77.90.31.11:4554 , piece_id=APWWSHCAFJKZEF5H3FDSJOXMXKKHNJEHO7CRXKYU3B3X4VDMBRZA, seg_pos={0 0}, upload=685543]: Long tail cancellation due to optimal threshold
Jan 7 07:10:33.985592638 (783h41m33.856675673s) [noise=true , address=77.90.31.19:4841 , piece_id=T4ZU4XYKFBN4Q4FCHIWPFGQKYUBRZFUY2LTNP57EB4RPIR3LCLWQ, seg_pos={0 0}, upload=685543]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF
Jan 7 07:10:33.985599121 (783h41m33.856682156s) [noise=true , address=77.90.31.19:4841 , piece_id=T4ZU4XYKFBN4Q4FCHIWPFGQKYUBRZFUY2LTNP57EB4RPIR3LCLWQ, seg_pos={0 0}, upload=685543]: Long tail cancellation due to optimal threshold
Jan 7 07:10:33.985754036 (783h41m33.856837071s) [noise=true , address=45.91.116.11:4117 , piece_id=Z5XRYSZPRFNSGBJCX66NZADXGM67M5Y7JDPMUWAS3KGFRZ3YTT4Q, seg_pos={0 0}, upload=685543]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF
Jan 7 07:10:33.985759727 (783h41m33.856842762s) [noise=true , address=45.91.116.11:4117 , piece_id=Z5XRYSZPRFNSGBJCX66NZADXGM67M5Y7JDPMUWAS3KGFRZ3YTT4Q, seg_pos={0 0}, upload=685543]: Long tail cancellation due to optimal threshold
Jan 7 07:10:33.985872070 (783h41m33.856955106s) [seg_pos={0 0}, upload=685543]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 7 07:10:33.985962032 (783h41m33.857045077s) [upload=685543]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 7 07:10:33.985966591 (783h41m33.857049636s) [upload=685543]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 7 07:10:34.275584985 (783h41m34.14666802s) [upload=685543]: Done uploading segments...
Jan 7 07:10:34.275603481 (783h41m34.146686516s) [upload=685543]: Upload finished. err: <nil>
Duration: 6.65102651s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [.........X........................................X........X] duration:
34.400838115s trace:
https://jaeger.dev.storj.io/trace/1ea01aa868761fdc
FAIL (2026-01-07 07:11:23 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: eestream: quiescence
======LOGS======
133194519 (783h41m50.004277565s) [noise=true , address=145.79.227.32:4851 , piece_id=5W74ZJOGSKONFPVTJJPHDWNFHLTWRU4KEVX25X4PQCKTRJZR3CZA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.133698059 (783h41m50.004781104s) [noise=true , address=145.79.227.33:4897 , piece_id=NLJJNXY2DGHMZWGJRSKGTYDVYZLYZJSVYVIZ35IUCMSI34RD5TBA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.134246032 (783h41m50.005329067s) [noise=true , address=145.79.227.44:4227 , piece_id=TCERBF6EX3JHAV63JYLUYA54OHYY33YEM25EQ6CIRS4746KNCULA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.134617359 (783h41m50.005700404s) [noise=true , address=145.79.227.30:4795 , piece_id=G7EGUQIODKUJDE6FEWO5B5C3RQYQ37SAHQYC6BI7X4JS6A7VMOXQ, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.135018602 (783h41m50.006101637s) [noise=true , address=77.90.31.16:4741 , piece_id=G4TPC522KLMJBYHYW2SYTUMJFVP4K54R63ZIHOCPQINQO5WE7Q2Q, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.135492385 (783h41m50.00657542s) [noise=true , address=145.79.227.41:4092 , piece_id=3WQEAVSBFU5KF6WVNGACEVQ6M42JEMWW6OHCXPCXVNZCOG4XSMGQ, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.135895753 (783h41m50.006978799s) [noise=true , address=145.79.227.34:4926 , piece_id=AUBI2PYIJK5YTKFZHL5WF7BQOKNKEPSTZQBERRGEOFZN7RE7T2SA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.136238677 (783h41m50.007321722s) [noise=true , address=193.149.164.143:4699 , piece_id=DFHC3CPJBT2U7MW5HSNGIGTPMD3QZD4ON62UJ2HNNVWRXOJ4A6PQ, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.136546201 (783h41m50.007629236s) [noise=true , address=145.79.227.36:4543 , piece_id=MUK6THUJD4QHK4VLL3R6DSZP47VV5RJAUPKG6RTPR7F6GTIC6GIA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.136565809 (783h41m50.007648844s) [noise=true , address=145.79.227.29:4946 , piece_id=FR3BLE7ANKJYAQG5GZJ4RL4543FQ55K7VHSJMQGCLC7UDRNKCASA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.137150974 (783h41m50.008234019s) [noise=true , address=145.79.227.35:4487 , piece_id=E5X6GYKRPSYHASZXZB62XA3LXB6EJPTMENDRW6BGEDVDUD5BW4TA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.137656586 (783h41m50.008739621s) [noise=true , address=77.90.31.11:4584 , piece_id=LIDDXUURY5ARSGQ6ETUFION36NJFWMIGJHOSOIE4ZUXJPTJNCGSA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.138273391 (783h41m50.009356436s) [noise=true , address=64.49.13.169:4713 , piece_id=6JKGRVL7KQOZVIIJM3OFQUHWUEF42KMW7Y6I5EPVKVRTKEQVBFNQ, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.138867362 (783h41m50.009950407s) [noise=true , address=64.49.13.155:4916 , piece_id=FPLKLM4RLSO54V7XX3YZF7CGXWSGKNBAKEYQ5FYFYHVE7CTS6PIA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.139440193 (783h41m50.010523238s) [noise=true , address=145.79.227.49:4372 , piece_id=HQ535NWA7OLGPNMYWPMHDJL3ICULP4GEXMYTKLDTC2SPMQYCRZOQ, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.139988797 (783h41m50.011071832s) [noise=true , address=45.91.116.10:4110 , piece_id=VODTQS7BAXPWLTDDTTW75FXYRDBGM2PSJQTIGCRCRNAMMTE55LJA, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:50.140847663 (783h41m50.011930698s) [noise=true , address=45.91.116.14:4200 , piece_id=ON5HNVYRNJTJCNV4JU6NI2ITS2VKZNNMYOBZUZQ7OLXPYAUQL2JQ, seg_pos={0 0}, upload=685546]: Uploading piece...
Jan 7 07:10:51.568606354 (783h41m51.439689389s) [upload=685546]: Next returned nil segment
Jan 7 07:10:51.568621312 (783h41m51.439704347s) [upload=685546]: Waiting for error group managing segments...
Jan 7 07:10:51.990916323 (783h41m51.861999368s) [noise=true , address=64.49.13.155:4916 , piece_id=FPLKLM4RLSO54V7XX3YZF7CGXWSGKNBAKEYQ5FYFYHVE7CTS6PIA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:51.994944574 (783h41m51.866027619s) [noise=true , address=145.79.227.44:4227 , piece_id=TCERBF6EX3JHAV63JYLUYA54OHYY33YEM25EQ6CIRS4746KNCULA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.000551040 (783h41m51.871634105s) [noise=true , address=145.79.227.41:4092 , piece_id=3WQEAVSBFU5KF6WVNGACEVQ6M42JEMWW6OHCXPCXVNZCOG4XSMGQ, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.005752454 (783h41m51.876835509s) [noise=true , address=145.79.227.30:4795 , piece_id=G7EGUQIODKUJDE6FEWO5B5C3RQYQ37SAHQYC6BI7X4JS6A7VMOXQ, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.006180148 (783h41m51.877263203s) [noise=true , address=145.79.227.32:4851 , piece_id=5W74ZJOGSKONFPVTJJPHDWNFHLTWRU4KEVX25X4PQCKTRJZR3CZA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.062554132 (783h41m51.933637177s) [noise=true , address=64.49.13.169:4713 , piece_id=6JKGRVL7KQOZVIIJM3OFQUHWUEF42KMW7Y6I5EPVKVRTKEQVBFNQ, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.070238635 (783h41m51.94132167s) [noise=true , address=145.79.227.34:4926 , piece_id=AUBI2PYIJK5YTKFZHL5WF7BQOKNKEPSTZQBERRGEOFZN7RE7T2SA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.088972997 (783h41m51.960056032s) [noise=true , address=145.79.227.33:4897 , piece_id=NLJJNXY2DGHMZWGJRSKGTYDVYZLYZJSVYVIZ35IUCMSI34RD5TBA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.281595960 (783h41m52.152678995s) [noise=true , address=77.90.31.12:4603 , piece_id=HPBI4VUKMT2KFKRW2JTX32HB2YGDGLB2V4BFRVVQ2725Y2IC2LOA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.356241606 (783h41m52.227324641s) [noise=true , address=77.90.31.11:4584 , piece_id=LIDDXUURY5ARSGQ6ETUFION36NJFWMIGJHOSOIE4ZUXJPTJNCGSA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.679275149 (783h41m52.550358204s) [noise=true , address=45.91.116.14:4200 , piece_id=ON5HNVYRNJTJCNV4JU6NI2ITS2VKZNNMYOBZUZQ7OLXPYAUQL2JQ, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:52.848093704 (783h41m52.719176749s) [noise=true , address=193.149.164.143:4699 , piece_id=DFHC3CPJBT2U7MW5HSNGIGTPMD3QZD4ON62UJ2HNNVWRXOJ4A6PQ, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:53.022732271 (783h41m52.893815316s) [noise=true , address=145.79.227.49:4372 , piece_id=HQ535NWA7OLGPNMYWPMHDJL3ICULP4GEXMYTKLDTC2SPMQYCRZOQ, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:53.048522630 (783h41m52.919605675s) [noise=true , address=145.79.227.36:4543 , piece_id=MUK6THUJD4QHK4VLL3R6DSZP47VV5RJAUPKG6RTPR7F6GTIC6GIA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: <nil>
Jan 7 07:10:53.048547036 (783h41m52.919630082s) [seg_pos={0 0}, upload=685546]: Segment reached optimal threshold of 14 pieces.
Jan 7 07:10:53.048761935 (783h41m52.919844971s) [noise=true , address=77.90.31.16:4741 , piece_id=G4TPC522KLMJBYHYW2SYTUMJFVP4K54R63ZIHOCPQINQO5WE7Q2Q, seg_pos={0 0}, upload=685546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12G8MCviRyA85FwyuN3sXF9pwWQnP6mki6Ux68Eefx42BeWUcNR): protocol: expected piece hash; context canceled; EOF
Jan 7 07:10:53.048771714 (783h41m52.919854749s) [noise=true , address=77.90.31.16:4741 , piece_id=G4TPC522KLMJBYHYW2SYTUMJFVP4K54R63ZIHOCPQINQO5WE7Q2Q, seg_pos={0 0}, upload=685546]: Long tail cancellation due to optimal threshold
Jan 7 07:10:53.049042570 (783h41m52.920125605s) [noise=true , address=145.79.227.29:4946 , piece_id=FR3BLE7ANKJYAQG5GZJ4RL4543FQ55K7VHSJMQGCLC7UDRNKCASA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12v9Q7aigjWPNXyshHmQt9YKrDu5af1vd1TxkDeHeKixSrWgrLX): protocol: expected piece hash; context canceled; EOF
Jan 7 07:10:53.049049483 (783h41m52.920132518s) [noise=true , address=145.79.227.29:4946 , piece_id=FR3BLE7ANKJYAQG5GZJ4RL4543FQ55K7VHSJMQGCLC7UDRNKCASA, seg_pos={0 0}, upload=685546]: Long tail cancellation due to optimal threshold
Jan 7 07:10:53.049146277 (783h41m52.920229322s) [noise=true , address=145.79.227.35:4487 , piece_id=E5X6GYKRPSYHASZXZB62XA3LXB6EJPTMENDRW6BGEDVDUD5BW4TA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1AdZLDV7UBbV2DimCHzmoNxycLigvLMx6Fy9U9QB35WTtiV1KX): protocol: expected piece hash; context canceled; EOF
Jan 7 07:10:53.049151166 (783h41m52.920234212s) [noise=true , address=145.79.227.35:4487 , piece_id=E5X6GYKRPSYHASZXZB62XA3LXB6EJPTMENDRW6BGEDVDUD5BW4TA, seg_pos={0 0}, upload=685546]: Long tail cancellation due to optimal threshold
Jan 7 07:10:53.049299398 (783h41m52.920382434s) [noise=true , address=45.91.116.10:4110 , piece_id=VODTQS7BAXPWLTDDTTW75FXYRDBGM2PSJQTIGCRCRNAMMTE55LJA, seg_pos={0 0}, upload=685546]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123KwusATu1ZKtNNVHZZZBE67ZEnVb8puyfQidBi3fEWREqdf1h): protocol: expected piece hash; context canceled; EOF
Jan 7 07:10:53.049308466 (783h41m52.920391512s) [noise=true , address=45.91.116.10:4110 , piece_id=VODTQS7BAXPWLTDDTTW75FXYRDBGM2PSJQTIGCRCRNAMMTE55LJA, seg_pos={0 0}, upload=685546]: Long tail cancellation due to optimal threshold
Jan 7 07:10:53.049324507 (783h41m52.920407542s) [seg_pos={0 0}, upload=685546]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 7 07:10:53.049388278 (783h41m52.920471313s) [upload=685546]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 7 07:10:53.049392767 (783h41m52.920475812s) [upload=685546]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 7 07:10:53.340338951 (783h41m53.211421996s) [upload=685546]: Done uploading segments...
Jan 7 07:10:53.340359300 (783h41m53.211442335s) [upload=685546]: Upload finished. err: <nil>
Duration: 34.400888731s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
49 out of 60 is failed (threshold=6) [XX.XXXXX.XXXXXX.XXXXXXXXXXX.XXXX.XXXX..XXXXX.XXXXX..XXXXX.XX] duration:
5.001175598s trace:
https://jaeger.dev.storj.io/trace/41dbcafe49587
FAIL (2026-01-07 07:11:11 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001189935s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [...........X.XXX.X.X..X....X.X.....X....X.X.....X.......X...] duration:
23.856789772s trace:
https://jaeger.dev.storj.io/trace/ae99c37afb2c1b6
FAIL (2026-01-07 07:11:08 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 23.856842513s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [...X..XX.X...X....X.........X..X.X..X.......XX.X....X....XX.] duration:
5.98093753s trace:
https://jaeger.dev.storj.io/trace/50ae9c48f3048fc3
FAIL (2026-01-07 07:11:06 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: SOAAPUTTLH3GV2TF2HUAN4BOA7OZKF4QICOHY4S5QEKESZD5N24A) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: BJHBHMTYA5HFEAKMPMM2PSNFBI7X32H2LP6VRY753L32SKGUKCSQ) context deadline exceeded
======LOGS======
Jan 7 07:10:31.248040589 (783h41m31.119123644s) [upload=685543]: Starting upload
Jan 7 07:10:31.248154456 (783h41m31.119237511s) [upload=685543]: Done starting upload
Jan 7 07:10:31.248536263 (783h41m31.119619308s) [upload=685543]: Uploading segments...
Jan 7 07:10:31.248606457 (783h41m31.119689502s) [upload=685543]: Got next segment. Inline: false
Jan 7 07:10:31.248647264 (783h41m31.119730309s) [upload=685543]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 7 07:10:31.248652915 (783h41m31.11973596s) [upload=685543]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 7 07:10:31.492628981 (783h41m31.363712026s) [seg_pos={0 0}, upload=685543]: Begin upload segment...
Jan 7 07:10:31.492693132 (783h41m31.363776177s) [seg_pos={0 0}, upload=685543]: Done begin upload segment.
Jan 7 07:10:31.492797992 (783h41m31.363881027s) [noise=true , address=145.79.227.45:4230 , piece_id=SQFIEICI2DR5H6UQM3D2JITSFG7NCZNBUMDVOUFNXPEAVYYG3UFQ, seg_pos={0 0}, upload=685543]: Uploading piece...
Jan 7 07:10:31.493578138 (783h41m31.364661183s) [noise=true , address=45.91.116.11:4117 , piece_id=Z5XRYSZPRFNSGBJCX66NZADXGM67M5Y7JDPMUWAS3KGFRZ3YTT4Q, seg_pos={0 0}, upload=685543]: Uploading piece...
Jan 7 07:10:31.494131151 (783h41m31.365214186s) [noise=true , address=145.79.227.37:4565 , piece_id=CPOFNSLYHUHQ4JCQMJEX24ZOBPJJCZYZZOAOSEREU4YDCIMLHEOQ, seg_pos={0 0}, upload=685543]: Uploading piece...
Jan 7 07:10:31.494488501 (783h41m31.365571546s) [noise=true , address=77.90.31.19:4841 , piece_id=T4ZU4XYKFBN4Q4FCHIWPFGQKYUBRZFUY2LTNP57EB4RPIR3LCLWQ, seg_pos={0 0}, upload=685543]: Uploading piece...
Jan 7 07:10:31.494776870 (783h41m31.365859915s) [noise=true , address=145.79.227.44:4195 , piece_id=DTUIBJ4JAWQGUOYOVRSMZEHQHBSZHYLISBYXKS22GEHDT7PO2CLA, seg_pos={0 0}, upload=685543]: Uploading piece...
Jan 7 07:10:31.495074447 (783h41m31.366157482s) [noise=true , address=145.79.227.12:4767 , piece_id=SRD2DTDKANNQNF4Q6H6D5QPYJU4SGL42ANBIDDB4AEICFTO6XERQ, seg_pos={0 0}, upload=685543]: Uploading piece...
Jan 7 07:10:31.495142487 (783h41m31.366225532s) [noise=true , address=77.90.31.16:4736 , piece_id=SOAAPUTTLH3GV2TF2HUAN4BOA7OZKF4QICOHY4S5QEKESZD5N24A, seg_pos={0 0}, upload=685543]: Uploading piece...
Jan 7 07:10:31.495649812 (783h41m31.366732847s) [noise=true , address=145.79.227.40:4050 , piece_id=EFGM7CQSPJJRM2ASQWDO6BUDKZ2KQBEGNIHLN4GNQ6SUWKVEEZ3Q, seg_pos={0 0}, upload=685543]: Uploading piece...
Jan 7 07:10:31.496148402 (783h41m31.367231447s) [noise=true , address=145.79.227.31:4799 , piece_id=BCSP53DP7YKGDPF5K37PZJQKD6TN7225D5RBFWOUO7OP4SBHMFBQ, seg_pos={0 0}, upload=685543]: Uploading piece...
Jan 7 07:10:31.496702267 (783h41m31.367785312s) [noise=true , address=77.90.31.11:4554 , piece_id=APWWSHCAFJKZEF5H3FDSJOXMXKKHNJEHO7CRXKYU3B3X4VDMBRZA, seg_pos={0 0}, upload=685543]: Uploading piece...
Jan 7 07:10:31.497256683 (783h41m31.368339728s) [noise=true , address=77.90.31.13:4629 , piece_id=TZJJYNZBZP5CNNLEC6TQHBN4XU4WQKMAHAPFJAUF5H2WTOK2DDCQ, seg_pos={0 0}, upload=685543]: Uploading piece...
Jan 7 07:10:31.499267872 (783h41m31.370350917s) [noise=true , address=145.79.227.26:4654 , piece_id=ODJVMCWVJGMYISGDRP4TLR4ABXXD5ZNUXKOSKDHV2HGVQLNKGD6Q, seg_pos={0 0}, upload=685543]: Uploading piece...
Jan 7 07:10:31.500996664 (783h41m31.372079709s) [noise=true , address=145.79.227.42:4121 , piece_id=RDI73ICBIBVAVGJABHW7TQRSUD6OYAMFZQJM7JLU2N7UT44OCSQQ, seg_pos={0 0}, upload=685543]: Uploading piece...
Jan 7 07:10:31.502637298 (783h41m31.373720333s) [noise=true , address=45.91.116.7:4010 , piece_id=BJHBHMTYA5HFEAKMPMM2PSNFBI7X32H2LP6VRY753L32SKGUKCSQ, seg_pos={0 0}, upload=685543]: Uploading piece...
Jan 7 07:10:31.504268755 (783h41m31.3753518s) [noise=true , address=145.79.227.6:4258 , piece_id=QWU6TB6IVDSS6H5AG4IMHOXPALH2AYZSEM36GTHY5KA6XPKCBXXQ, seg_pos={0 0}, upload=685543]: Uploading piece...
Jan 7 07:10:31.506042783 (783h41m31.377125818s) [noise=true , address=145.79.227.2:4113 , piece_id=YQ54OQ26JK34YPLN3TZXR5LWJOFN3RUVZJ5YUZXGI2RY2OKLRFSA, seg_pos={0 0}, upload=685543]: Uploading piece...
Jan 7 07:10:33.202789827 (783h41m33.073872872s) [upload=685543]: Next returned nil segment
Jan 7 07:10:33.202822870 (783h41m33.073905905s) [upload=685543]: Waiting for error group managing segments...
Jan 7 07:10:33.484280988 (783h41m33.355364023s) [noise=true , address=145.79.227.44:4195 , piece_id=DTUIBJ4JAWQGUOYOVRSMZEHQHBSZHYLISBYXKS22GEHDT7PO2CLA, seg_pos={0 0}, upload=685543]: Done uploading piece. err: <nil>
Jan 7 07:10:33.488021811 (783h41m33.359104846s) [noise=true , address=145.79.227.12:4767 , piece_id=SRD2DTDKANNQNF4Q6H6D5QPYJU4SGL42ANBIDDB4AEICFTO6XERQ, seg_pos={0 0}, upload=685543]: Done uploading piece. err: <nil>
Jan 7 07:10:33.491589184 (783h41m33.362672239s) [noise=true , address=145.79.227.40:4050 , piece_id=EFGM7CQSPJJRM2ASQWDO6BUDKZ2KQBEGNIHLN4GNQ6SUWKVEEZ3Q, seg_pos={0 0}, upload=685543]: Done uploading piece. err: <nil>
Jan 7 07:10:33.498955099 (783h41m33.370038134s) [noise=true , address=145.79.227.6:4258 , piece_id=QWU6TB6IVDSS6H5AG4IMHOXPALH2AYZSEM36GTHY5KA6XPKCBXXQ, seg_pos={0 0}, upload=685543]: Done uploading piece. err: <nil>
Jan 7 07:10:33.501171139 (783h41m33.372254174s) [noise=true , address=145.79.227.37:4565 , piece_id=CPOFNSLYHUHQ4JCQMJEX24ZOBPJJCZYZZOAOSEREU4YDCIMLHEOQ, seg_pos={0 0}, upload=685543]: Done uploading piece. err: <nil>
Jan 7 07:10:33.510816175 (783h41m33.38189922s) [noise=true , address=145.79.227.31:4799 , piece_id=BCSP53DP7YKGDPF5K37PZJQKD6TN7225D5RBFWOUO7OP4SBHMFBQ, seg_pos={0 0}, upload=685543]: Done uploading piece. err: <nil>
Jan 7 07:10:33.511084094 (783h41m33.382167129s) [noise=true , address=145.79.227.45:4230 , piece_id=SQFIEICI2DR5H6UQM3D2JITSFG7NCZNBUMDVOUFNXPEAVYYG3UFQ, seg_pos={0 0}, upload=685543]: Done uploading piece. err: <nil>
Jan 7 07:10:33.516566915 (783h41m33.38764997s) [noise=true , address=145.79.227.26:4654 , piece_id=ODJVMCWVJGMYISGDRP4TLR4ABXXD5ZNUXKOSKDHV2HGVQLNKGD6Q, seg_pos={0 0}, upload=685543]: Done uploading piece. err: <nil>
Jan 7 07:10:33.517421321 (783h41m33.388504357s) [noise=true , address=145.79.227.42:4121 , piece_id=RDI73ICBIBVAVGJABHW7TQRSUD6OYAMFZQJM7JLU2N7UT44OCSQQ, seg_pos={0 0}, upload=685543]: Done uploading piece. err: <nil>
Jan 7 07:10:33.518568827 (783h41m33.389651872s) [noise=true , address=145.79.227.2:4113 , piece_id=YQ54OQ26JK34YPLN3TZXR5LWJOFN3RUVZJ5YUZXGI2RY2OKLRFSA, seg_pos={0 0}, upload=685543]: Done uploading piece. err: <nil>
Jan 7 07:10:33.781941837 (783h41m33.653024882s) [noise=true , address=77.90.31.16:4736 , piece_id=SOAAPUTTLH3GV2TF2HUAN4BOA7OZKF4QICOHY4S5QEKESZD5N24A, seg_pos={0 0}, upload=685543]: Done uploading piece. err: <nil>
Jan 7 07:10:33.851158897 (783h41m33.722241932s) [noise=true , address=77.90.31.13:4629 , piece_id=TZJJYNZBZP5CNNLEC6TQHBN4XU4WQKMAHAPFJAUF5H2WTOK2DDCQ, seg_pos={0 0}, upload=685543]: Done uploading piece. err: <nil>
Jan 7 07:10:33.985266949 (783h41m33.856349984s) [noise=true , address=45.91.116.7:4010 , piece_id=BJHBHMTYA5HFEAKMPMM2PSNFBI7X32H2LP6VRY753L32SKGUKCSQ, seg_pos={0 0}, upload=685543]: Done uploading piece. err: <nil>
Jan 7 07:10:33.985294601 (783h41m33.856377647s) [seg_pos={0 0}, upload=685543]: Segment reached optimal threshold of 13 pieces.
Jan 7 07:10:33.985506545 (783h41m33.85658959s) [noise=true , address=77.90.31.11:4554 , piece_id=APWWSHCAFJKZEF5H3FDSJOXMXKKHNJEHO7CRXKYU3B3X4VDMBRZA, seg_pos={0 0}, upload=685543]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: write tcp 188.166.225.170:43140->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:43140->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43140->77.90.31.11:4554: use of closed network connection
Jan 7 07:10:33.985515993 (783h41m33.856599028s) [noise=true , address=77.90.31.11:4554 , piece_id=APWWSHCAFJKZEF5H3FDSJOXMXKKHNJEHO7CRXKYU3B3X4VDMBRZA, seg_pos={0 0}, upload=685543]: Long tail cancellation due to optimal threshold
Jan 7 07:10:33.985592638 (783h41m33.856675673s) [noise=true , address=77.90.31.19:4841 , piece_id=T4ZU4XYKFBN4Q4FCHIWPFGQKYUBRZFUY2LTNP57EB4RPIR3LCLWQ, seg_pos={0 0}, upload=685543]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF
Jan 7 07:10:33.985599121 (783h41m33.856682156s) [noise=true , address=77.90.31.19:4841 , piece_id=T4ZU4XYKFBN4Q4FCHIWPFGQKYUBRZFUY2LTNP57EB4RPIR3LCLWQ, seg_pos={0 0}, upload=685543]: Long tail cancellation due to optimal threshold
Jan 7 07:10:33.985754036 (783h41m33.856837071s) [noise=true , address=45.91.116.11:4117 , piece_id=Z5XRYSZPRFNSGBJCX66NZADXGM67M5Y7JDPMUWAS3KGFRZ3YTT4Q, seg_pos={0 0}, upload=685543]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF
Jan 7 07:10:33.985759727 (783h41m33.856842762s) [noise=true , address=45.91.116.11:4117 , piece_id=Z5XRYSZPRFNSGBJCX66NZADXGM67M5Y7JDPMUWAS3KGFRZ3YTT4Q, seg_pos={0 0}, upload=685543]: Long tail cancellation due to optimal threshold
Jan 7 07:10:33.985872070 (783h41m33.856955106s) [seg_pos={0 0}, upload=685543]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 7 07:10:33.985962032 (783h41m33.857045077s) [upload=685543]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 7 07:10:33.985966591 (783h41m33.857049636s) [upload=685543]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 7 07:10:34.275584985 (783h41m34.14666802s) [upload=685543]: Done uploading segments...
Jan 7 07:10:34.275603481 (783h41m34.146686516s) [upload=685543]: Upload finished. err: <nil>
Duration: 5.980951317s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [....X..XX.X...X....X.........X..X.X..X.......XX.X....X....XX] duration:
30.003582071s trace:
https://jaeger.dev.storj.io/trace/50ae9c48f3048fc3
FAIL (2026-01-07 07:11:00 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: SOAAPUTTLH3GV2TF2HUAN4BOA7OZKF4QICOHY4S5QEKESZD5N24A) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: BJHBHMTYA5HFEAKMPMM2PSNFBI7X32H2LP6VRY753L32SKGUKCSQ) context deadline exceeded
======LOGS======
Jan 7 07:10:31.248040589 (783h41m31.119123644s) [upload=685543]: Starting upload
Jan 7 07:10:31.248154456 (783h41m31.119237511s) [upload=685543]: Done starting upload
Jan 7 07:10:31.248536263 (783h41m31.119619308s) [upload=685543]: Uploading segments...
Jan 7 07:10:31.248606457 (783h41m31.119689502s) [upload=685543]: Got next segment. Inline: false
Jan 7 07:10:31.248647264 (783h41m31.119730309s) [upload=685543]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 7 07:10:31.248652915 (783h41m31.11973596s) [upload=685543]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 7 07:10:31.492628981 (783h41m31.363712026s) [seg_pos={0 0}, upload=685543]: Begin upload segment...
Jan 7 07:10:31.492693132 (783h41m31.363776177s) [seg_pos={0 0}, upload=685543]: Done begin upload segment.
Jan 7 07:10:31.492797992 (783h41m31.363881027s) [noise=true , address=145.79.227.45:4230 , piece_id=SQFIEICI2DR5H6UQM3D2JITSFG7NCZNBUMDVOUFNXPEAVYYG3UFQ, seg_pos={0 0}, upload=685543]: Uploading piece...
Jan 7 07:10:31.493578138 (783h41m31.364661183s) [noise=true , address=45.91.116.11:4117 , piece_id=Z5XRYSZPRFNSGBJCX66NZADXGM67M5Y7JDPMUWAS3KGFRZ3YTT4Q, seg_pos={0 0}, upload=685543]: Uploading piece...
Jan 7 07:10:31.494131151 (783h41m31.365214186s) [noise=true , address=145.79.227.37:4565 , piece_id=CPOFNSLYHUHQ4JCQMJEX24ZOBPJJCZYZZOAOSEREU4YDCIMLHEOQ, seg_pos={0 0}, upload=685543]: Uploading piece...
Jan 7 07:10:31.494488501 (783h41m31.365571546s) [noise=true , address=77.90.31.19:4841 , piece_id=T4ZU4XYKFBN4Q4FCHIWPFGQKYUBRZFUY2LTNP57EB4RPIR3LCLWQ, seg_pos={0 0}, upload=685543]: Uploading piece...
Jan 7 07:10:31.494776870 (783h41m31.365859915s) [noise=true , address=145.79.227.44:4195 , piece_id=DTUIBJ4JAWQGUOYOVRSMZEHQHBSZHYLISBYXKS22GEHDT7PO2CLA, seg_pos={0 0}, upload=685543]: Uploading piece...
Jan 7 07:10:31.495074447 (783h41m31.366157482s) [noise=true , address=145.79.227.12:4767 , piece_id=SRD2DTDKANNQNF4Q6H6D5QPYJU4SGL42ANBIDDB4AEICFTO6XERQ, seg_pos={0 0}, upload=685543]: Uploading piece...
Jan 7 07:10:31.495142487 (783h41m31.366225532s) [noise=true , address=77.90.31.16:4736 , piece_id=SOAAPUTTLH3GV2TF2HUAN4BOA7OZKF4QICOHY4S5QEKESZD5N24A, seg_pos={0 0}, upload=685543]: Uploading piece...
Jan 7 07:10:31.495649812 (783h41m31.366732847s) [noise=true , address=145.79.227.40:4050 , piece_id=EFGM7CQSPJJRM2ASQWDO6BUDKZ2KQBEGNIHLN4GNQ6SUWKVEEZ3Q, seg_pos={0 0}, upload=685543]: Uploading piece...
Jan 7 07:10:31.496148402 (783h41m31.367231447s) [noise=true , address=145.79.227.31:4799 , piece_id=BCSP53DP7YKGDPF5K37PZJQKD6TN7225D5RBFWOUO7OP4SBHMFBQ, seg_pos={0 0}, upload=685543]: Uploading piece...
Jan 7 07:10:31.496702267 (783h41m31.367785312s) [noise=true , address=77.90.31.11:4554 , piece_id=APWWSHCAFJKZEF5H3FDSJOXMXKKHNJEHO7CRXKYU3B3X4VDMBRZA, seg_pos={0 0}, upload=685543]: Uploading piece...
Jan 7 07:10:31.497256683 (783h41m31.368339728s) [noise=true , address=77.90.31.13:4629 , piece_id=TZJJYNZBZP5CNNLEC6TQHBN4XU4WQKMAHAPFJAUF5H2WTOK2DDCQ, seg_pos={0 0}, upload=685543]: Uploading piece...
Jan 7 07:10:31.499267872 (783h41m31.370350917s) [noise=true , address=145.79.227.26:4654 , piece_id=ODJVMCWVJGMYISGDRP4TLR4ABXXD5ZNUXKOSKDHV2HGVQLNKGD6Q, seg_pos={0 0}, upload=685543]: Uploading piece...
Jan 7 07:10:31.500996664 (783h41m31.372079709s) [noise=true , address=145.79.227.42:4121 , piece_id=RDI73ICBIBVAVGJABHW7TQRSUD6OYAMFZQJM7JLU2N7UT44OCSQQ, seg_pos={0 0}, upload=685543]: Uploading piece...
Jan 7 07:10:31.502637298 (783h41m31.373720333s) [noise=true , address=45.91.116.7:4010 , piece_id=BJHBHMTYA5HFEAKMPMM2PSNFBI7X32H2LP6VRY753L32SKGUKCSQ, seg_pos={0 0}, upload=685543]: Uploading piece...
Jan 7 07:10:31.504268755 (783h41m31.3753518s) [noise=true , address=145.79.227.6:4258 , piece_id=QWU6TB6IVDSS6H5AG4IMHOXPALH2AYZSEM36GTHY5KA6XPKCBXXQ, seg_pos={0 0}, upload=685543]: Uploading piece...
Jan 7 07:10:31.506042783 (783h41m31.377125818s) [noise=true , address=145.79.227.2:4113 , piece_id=YQ54OQ26JK34YPLN3TZXR5LWJOFN3RUVZJ5YUZXGI2RY2OKLRFSA, seg_pos={0 0}, upload=685543]: Uploading piece...
Jan 7 07:10:33.202789827 (783h41m33.073872872s) [upload=685543]: Next returned nil segment
Jan 7 07:10:33.202822870 (783h41m33.073905905s) [upload=685543]: Waiting for error group managing segments...
Jan 7 07:10:33.484280988 (783h41m33.355364023s) [noise=true , address=145.79.227.44:4195 , piece_id=DTUIBJ4JAWQGUOYOVRSMZEHQHBSZHYLISBYXKS22GEHDT7PO2CLA, seg_pos={0 0}, upload=685543]: Done uploading piece. err: <nil>
Jan 7 07:10:33.488021811 (783h41m33.359104846s) [noise=true , address=145.79.227.12:4767 , piece_id=SRD2DTDKANNQNF4Q6H6D5QPYJU4SGL42ANBIDDB4AEICFTO6XERQ, seg_pos={0 0}, upload=685543]: Done uploading piece. err: <nil>
Jan 7 07:10:33.491589184 (783h41m33.362672239s) [noise=true , address=145.79.227.40:4050 , piece_id=EFGM7CQSPJJRM2ASQWDO6BUDKZ2KQBEGNIHLN4GNQ6SUWKVEEZ3Q, seg_pos={0 0}, upload=685543]: Done uploading piece. err: <nil>
Jan 7 07:10:33.498955099 (783h41m33.370038134s) [noise=true , address=145.79.227.6:4258 , piece_id=QWU6TB6IVDSS6H5AG4IMHOXPALH2AYZSEM36GTHY5KA6XPKCBXXQ, seg_pos={0 0}, upload=685543]: Done uploading piece. err: <nil>
Jan 7 07:10:33.501171139 (783h41m33.372254174s) [noise=true , address=145.79.227.37:4565 , piece_id=CPOFNSLYHUHQ4JCQMJEX24ZOBPJJCZYZZOAOSEREU4YDCIMLHEOQ, seg_pos={0 0}, upload=685543]: Done uploading piece. err: <nil>
Jan 7 07:10:33.510816175 (783h41m33.38189922s) [noise=true , address=145.79.227.31:4799 , piece_id=BCSP53DP7YKGDPF5K37PZJQKD6TN7225D5RBFWOUO7OP4SBHMFBQ, seg_pos={0 0}, upload=685543]: Done uploading piece. err: <nil>
Jan 7 07:10:33.511084094 (783h41m33.382167129s) [noise=true , address=145.79.227.45:4230 , piece_id=SQFIEICI2DR5H6UQM3D2JITSFG7NCZNBUMDVOUFNXPEAVYYG3UFQ, seg_pos={0 0}, upload=685543]: Done uploading piece. err: <nil>
Jan 7 07:10:33.516566915 (783h41m33.38764997s) [noise=true , address=145.79.227.26:4654 , piece_id=ODJVMCWVJGMYISGDRP4TLR4ABXXD5ZNUXKOSKDHV2HGVQLNKGD6Q, seg_pos={0 0}, upload=685543]: Done uploading piece. err: <nil>
Jan 7 07:10:33.517421321 (783h41m33.388504357s) [noise=true , address=145.79.227.42:4121 , piece_id=RDI73ICBIBVAVGJABHW7TQRSUD6OYAMFZQJM7JLU2N7UT44OCSQQ, seg_pos={0 0}, upload=685543]: Done uploading piece. err: <nil>
Jan 7 07:10:33.518568827 (783h41m33.389651872s) [noise=true , address=145.79.227.2:4113 , piece_id=YQ54OQ26JK34YPLN3TZXR5LWJOFN3RUVZJ5YUZXGI2RY2OKLRFSA, seg_pos={0 0}, upload=685543]: Done uploading piece. err: <nil>
Jan 7 07:10:33.781941837 (783h41m33.653024882s) [noise=true , address=77.90.31.16:4736 , piece_id=SOAAPUTTLH3GV2TF2HUAN4BOA7OZKF4QICOHY4S5QEKESZD5N24A, seg_pos={0 0}, upload=685543]: Done uploading piece. err: <nil>
Jan 7 07:10:33.851158897 (783h41m33.722241932s) [noise=true , address=77.90.31.13:4629 , piece_id=TZJJYNZBZP5CNNLEC6TQHBN4XU4WQKMAHAPFJAUF5H2WTOK2DDCQ, seg_pos={0 0}, upload=685543]: Done uploading piece. err: <nil>
Jan 7 07:10:33.985266949 (783h41m33.856349984s) [noise=true , address=45.91.116.7:4010 , piece_id=BJHBHMTYA5HFEAKMPMM2PSNFBI7X32H2LP6VRY753L32SKGUKCSQ, seg_pos={0 0}, upload=685543]: Done uploading piece. err: <nil>
Jan 7 07:10:33.985294601 (783h41m33.856377647s) [seg_pos={0 0}, upload=685543]: Segment reached optimal threshold of 13 pieces.
Jan 7 07:10:33.985506545 (783h41m33.85658959s) [noise=true , address=77.90.31.11:4554 , piece_id=APWWSHCAFJKZEF5H3FDSJOXMXKKHNJEHO7CRXKYU3B3X4VDMBRZA, seg_pos={0 0}, upload=685543]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: write tcp 188.166.225.170:43140->77.90.31.11:4554: use of closed network connection; write tcp 188.166.225.170:43140->77.90.31.11:4554: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43140->77.90.31.11:4554: use of closed network connection
Jan 7 07:10:33.985515993 (783h41m33.856599028s) [noise=true , address=77.90.31.11:4554 , piece_id=APWWSHCAFJKZEF5H3FDSJOXMXKKHNJEHO7CRXKYU3B3X4VDMBRZA, seg_pos={0 0}, upload=685543]: Long tail cancellation due to optimal threshold
Jan 7 07:10:33.985592638 (783h41m33.856675673s) [noise=true , address=77.90.31.19:4841 , piece_id=T4ZU4XYKFBN4Q4FCHIWPFGQKYUBRZFUY2LTNP57EB4RPIR3LCLWQ, seg_pos={0 0}, upload=685543]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA): protocol: expected piece hash; context canceled; EOF
Jan 7 07:10:33.985599121 (783h41m33.856682156s) [noise=true , address=77.90.31.19:4841 , piece_id=T4ZU4XYKFBN4Q4FCHIWPFGQKYUBRZFUY2LTNP57EB4RPIR3LCLWQ, seg_pos={0 0}, upload=685543]: Long tail cancellation due to optimal threshold
Jan 7 07:10:33.985754036 (783h41m33.856837071s) [noise=true , address=45.91.116.11:4117 , piece_id=Z5XRYSZPRFNSGBJCX66NZADXGM67M5Y7JDPMUWAS3KGFRZ3YTT4Q, seg_pos={0 0}, upload=685543]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF
Jan 7 07:10:33.985759727 (783h41m33.856842762s) [noise=true , address=45.91.116.11:4117 , piece_id=Z5XRYSZPRFNSGBJCX66NZADXGM67M5Y7JDPMUWAS3KGFRZ3YTT4Q, seg_pos={0 0}, upload=685543]: Long tail cancellation due to optimal threshold
Jan 7 07:10:33.985872070 (783h41m33.856955106s) [seg_pos={0 0}, upload=685543]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 7 07:10:33.985962032 (783h41m33.857045077s) [upload=685543]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 7 07:10:33.985966591 (783h41m33.857049636s) [upload=685543]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 7 07:10:34.275584985 (783h41m34.14666802s) [upload=685543]: Done uploading segments...
Jan 7 07:10:34.275603481 (783h41m34.146686516s) [upload=685543]: Upload finished. err: <nil>
Duration: 30.00360794s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [.X....X..............X...............X...................X..] duration:
19.891461618s trace:
https://jaeger.dev.storj.io/trace/36161e151ba11c34
FAIL (2026-01-07 07:10:51 UTC)
orbiter warning: context deadline exceeded
Duration: 19.891492626s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
2 out of 60 is failed (threshold=3) [..........X........................................X........] duration:
47.523530262s trace:
https://jaeger.dev.storj.io/trace/dd40b8135c8dd51
FAIL (2026-01-07 07:10:33 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
8.476816160 (783h34m38.347899195s) [noise=true , address=145.79.227.40:4081 , piece_id=4546UK2SYRRCRNEBULDXXER6XCFYFP2ZDR3M4FAKD623IOOXNDFQ, seg_pos={0 0}, upload=685442]: Uploading piece...
Jan 7 07:03:38.476866375 (783h34m38.34794941s) [noise=true , address=145.79.227.36:4520 , piece_id=XGREKISAZVBCEOENVXW42EL2JLUSRPPP6SB5KUQ2HINPUBLKWEDQ, seg_pos={0 0}, upload=685442]: Uploading piece...
Jan 7 07:03:38.477291164 (783h34m38.348374209s) [noise=true , address=145.79.227.35:4502 , piece_id=ICYALD3UZQWQEEFLWWVVCJM4PBZRYVEWR6JGPLW5VFE53QSTCVMA, seg_pos={0 0}, upload=685442]: Uploading piece...
Jan 7 07:03:38.477661359 (783h34m38.348744404s) [noise=true , address=145.79.227.30:4777 , piece_id=IIH7FF62RDO5PJBQANPP7A76XQZV67FWGFBV6UNO6CHDYIYVDJCA, seg_pos={0 0}, upload=685442]: Uploading piece...
Jan 7 07:03:38.478069887 (783h34m38.349152932s) [noise=true , address=77.90.31.19:4865 , piece_id=4RSRZMKEYTMSKHCPWUDNOEGPHI6DOOYGVJU6GZXVO4SL6754QP6Q, seg_pos={0 0}, upload=685442]: Uploading piece...
Jan 7 07:03:38.478406517 (783h34m38.349489562s) [noise=true , address=145.79.227.32:4833 , piece_id=AWCHK4XJ66XON7FXPKB6NLYMNZYHFJKDLEUJCCAXGQP5PV2ID3AQ, seg_pos={0 0}, upload=685442]: Uploading piece...
Jan 7 07:03:38.478798054 (783h34m38.349881099s) [noise=true , address=64.49.13.169:4706 , piece_id=R2UIMDRHGFOECEZLJA3VVWCGFAU3GIZCU4FXD4JGPVW4P3YE76GA, seg_pos={0 0}, upload=685442]: Uploading piece...
Jan 7 07:03:38.479188116 (783h34m38.350271161s) [noise=true , address=145.79.227.29:4971 , piece_id=QJVVRKRFQPJLCSWIZQ5O4BKUALADPFSCIQ3X46KZRI7Q4VQOWZ4A, seg_pos={0 0}, upload=685442]: Uploading piece...
Jan 7 07:03:38.479554994 (783h34m38.350638039s) [noise=true , address=145.79.227.44:4211 , piece_id=AJHEFGJN3LI6MKKVRYIJDKZV2OG7UKY6MJ2NTAH5GELSIKE5WHRA, seg_pos={0 0}, upload=685442]: Uploading piece...
Jan 7 07:03:38.480021573 (783h34m38.351104618s) [noise=true , address=145.79.227.33:4886 , piece_id=3HKNOFKPTMSGYVCKNEL25Y5JURZLGWNFCHHE2IDGSZARQHSSA7OA, seg_pos={0 0}, upload=685442]: Uploading piece...
Jan 7 07:03:38.480525282 (783h34m38.351608317s) [noise=true , address=77.90.31.11:4587 , piece_id=VLFVPF2Y45LIAH2P2LCURQ6H2W2NVK3JSTU6EAAWL4ROD6W3VA2A, seg_pos={0 0}, upload=685442]: Uploading piece...
Jan 7 07:03:38.481874692 (783h34m38.352957737s) [noise=true , address=145.79.227.43:4163 , piece_id=ZRMQPLKZBQ5LW54C5IELYFSGLFUXROIHLLP4KRBJTG6Q6IVBEBZQ, seg_pos={0 0}, upload=685442]: Uploading piece...
Jan 7 07:03:38.483214132 (783h34m38.354297167s) [noise=true , address=145.79.227.34:4939 , piece_id=XFKY5SDSPO7YW7RISI5ENW353T2XTSBKNXKORXJSFYFNOOOEHCUQ, seg_pos={0 0}, upload=685442]: Uploading piece...
Jan 7 07:03:38.484535869 (783h34m38.355618904s) [noise=true , address=77.90.31.15:4717 , piece_id=5IVMKFZ3DHM3VWRV4O7BNPOIMLNWNSZDI3F4MALJSSEFYIRQDNMA, seg_pos={0 0}, upload=685442]: Uploading piece...
Jan 7 07:03:38.485917810 (783h34m38.357000855s) [noise=true , address=77.90.31.14:4684 , piece_id=A2WJQXJZB3KUVU6QIUJXIK33ITV3W4TS4P6OYU577HQ5P6UQXFXA, seg_pos={0 0}, upload=685442]: Uploading piece...
Jan 7 07:03:38.487207396 (783h34m38.358290431s) [noise=true , address=45.91.116.11:4138 , piece_id=M4ORVHQAZE5G7VKS4D32PFKJZ3GH4GOD44J7AFIOXBQSIYKDARDQ, seg_pos={0 0}, upload=685442]: Uploading piece...
Jan 7 07:03:38.488628252 (783h34m38.359711297s) [noise=true , address=193.149.164.143:4696 , piece_id=ANKNUEKVS4EY64ALVJVYQJ6I3JWCYJKGDDRTZ26MDB3GZHDK6QIQ, seg_pos={0 0}, upload=685442]: Uploading piece...
Jan 7 07:03:39.906410053 (783h34m39.777493098s) [upload=685442]: Next returned nil segment
Jan 7 07:03:39.906433628 (783h34m39.777516663s) [upload=685442]: Waiting for error group managing segments...
Jan 7 07:03:40.341600729 (783h34m40.212683774s) [noise=true , address=145.79.227.34:4939 , piece_id=XFKY5SDSPO7YW7RISI5ENW353T2XTSBKNXKORXJSFYFNOOOEHCUQ, seg_pos={0 0}, upload=685442]: Done uploading piece. err: <nil>
Jan 7 07:03:40.343637298 (783h34m40.214720333s) [noise=true , address=145.79.227.43:4163 , piece_id=ZRMQPLKZBQ5LW54C5IELYFSGLFUXROIHLLP4KRBJTG6Q6IVBEBZQ, seg_pos={0 0}, upload=685442]: Done uploading piece. err: <nil>
Jan 7 07:03:40.351048580 (783h34m40.222131625s) [noise=true , address=64.49.13.169:4706 , piece_id=R2UIMDRHGFOECEZLJA3VVWCGFAU3GIZCU4FXD4JGPVW4P3YE76GA, seg_pos={0 0}, upload=685442]: Done uploading piece. err: <nil>
Jan 7 07:03:40.353418884 (783h34m40.224501919s) [noise=true , address=145.79.227.44:4211 , piece_id=AJHEFGJN3LI6MKKVRYIJDKZV2OG7UKY6MJ2NTAH5GELSIKE5WHRA, seg_pos={0 0}, upload=685442]: Done uploading piece. err: <nil>
Jan 7 07:03:40.365409475 (783h34m40.23649253s) [noise=true , address=145.79.227.30:4777 , piece_id=IIH7FF62RDO5PJBQANPP7A76XQZV67FWGFBV6UNO6CHDYIYVDJCA, seg_pos={0 0}, upload=685442]: Done uploading piece. err: <nil>
Jan 7 07:03:40.424692969 (783h34m40.295776014s) [noise=true , address=145.79.227.40:4081 , piece_id=4546UK2SYRRCRNEBULDXXER6XCFYFP2ZDR3M4FAKD623IOOXNDFQ, seg_pos={0 0}, upload=685442]: Done uploading piece. err: <nil>
Jan 7 07:03:40.640456525 (783h34m40.51153957s) [noise=true , address=77.90.31.14:4684 , piece_id=A2WJQXJZB3KUVU6QIUJXIK33ITV3W4TS4P6OYU577HQ5P6UQXFXA, seg_pos={0 0}, upload=685442]: Done uploading piece. err: <nil>
Jan 7 07:03:40.660090451 (783h34m40.531173496s) [noise=true , address=77.90.31.11:4587 , piece_id=VLFVPF2Y45LIAH2P2LCURQ6H2W2NVK3JSTU6EAAWL4ROD6W3VA2A, seg_pos={0 0}, upload=685442]: Done uploading piece. err: <nil>
Jan 7 07:03:40.704654687 (783h34m40.575737723s) [noise=true , address=77.90.31.19:4865 , piece_id=4RSRZMKEYTMSKHCPWUDNOEGPHI6DOOYGVJU6GZXVO4SL6754QP6Q, seg_pos={0 0}, upload=685442]: Done uploading piece. err: <nil>
Jan 7 07:03:40.754800282 (783h34m40.625883317s) [noise=true , address=145.79.227.32:4833 , piece_id=AWCHK4XJ66XON7FXPKB6NLYMNZYHFJKDLEUJCCAXGQP5PV2ID3AQ, seg_pos={0 0}, upload=685442]: Done uploading piece. err: <nil>
Jan 7 07:03:40.770729153 (783h34m40.641812188s) [noise=true , address=45.91.116.11:4138 , piece_id=M4ORVHQAZE5G7VKS4D32PFKJZ3GH4GOD44J7AFIOXBQSIYKDARDQ, seg_pos={0 0}, upload=685442]: Done uploading piece. err: <nil>
Jan 7 07:03:41.178899310 (783h34m41.049982345s) [noise=true , address=45.91.116.14:4210 , piece_id=LP4YQNRLQ7HGB3SFI2YPQ34E7RXZJ3NNJAKJ35O33FZ3SL2DXFIA, seg_pos={0 0}, upload=685442]: Done uploading piece. err: <nil>
Jan 7 07:03:41.195846850 (783h34m41.066929885s) [noise=true , address=193.149.164.143:4696 , piece_id=ANKNUEKVS4EY64ALVJVYQJ6I3JWCYJKGDDRTZ26MDB3GZHDK6QIQ, seg_pos={0 0}, upload=685442]: Done uploading piece. err: <nil>
Jan 7 07:03:41.673145746 (783h34m41.544228781s) [noise=true , address=145.79.227.35:4502 , piece_id=ICYALD3UZQWQEEFLWWVVCJM4PBZRYVEWR6JGPLW5VFE53QSTCVMA, seg_pos={0 0}, upload=685442]: Done uploading piece. err: <nil>
Jan 7 07:03:41.673168520 (783h34m41.544251555s) [seg_pos={0 0}, upload=685442]: Segment reached optimal threshold of 14 pieces.
Jan 7 07:03:41.673410881 (783h34m41.544493916s) [noise=true , address=145.79.227.29:4971 , piece_id=QJVVRKRFQPJLCSWIZQ5O4BKUALADPFSCIQ3X46KZRI7Q4VQOWZ4A, seg_pos={0 0}, upload=685442]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1pYDfLBXpGNfJsH2sNUSSuKPTtgXNUEZx1oGB4SxzsFwdX4UiH): protocol: expected piece hash; context canceled; EOF
Jan 7 07:03:41.673419758 (783h34m41.544502793s) [noise=true , address=145.79.227.29:4971 , piece_id=QJVVRKRFQPJLCSWIZQ5O4BKUALADPFSCIQ3X46KZRI7Q4VQOWZ4A, seg_pos={0 0}, upload=685442]: Long tail cancellation due to optimal threshold
Jan 7 07:03:41.673445287 (783h34m41.544528332s) [noise=true , address=145.79.227.33:4886 , piece_id=3HKNOFKPTMSGYVCKNEL25Y5JURZLGWNFCHHE2IDGSZARQHSSA7OA, seg_pos={0 0}, upload=685442]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR): protocol: expected piece hash; context canceled; EOF
Jan 7 07:03:41.673455586 (783h34m41.544538631s) [noise=true , address=145.79.227.33:4886 , piece_id=3HKNOFKPTMSGYVCKNEL25Y5JURZLGWNFCHHE2IDGSZARQHSSA7OA, seg_pos={0 0}, upload=685442]: Long tail cancellation due to optimal threshold
Jan 7 07:03:41.673597717 (783h34m41.544680782s) [noise=true , address=77.90.31.15:4717 , piece_id=5IVMKFZ3DHM3VWRV4O7BNPOIMLNWNSZDI3F4MALJSSEFYIRQDNMA, seg_pos={0 0}, upload=685442]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ULANtxKNaJviAZFsmHjDQWjG4x2wfmjCwbQtHmn9LpnELmkPc): protocol: expected piece hash; context canceled; EOF
Jan 7 07:03:41.673615792 (783h34m41.544698837s) [noise=true , address=77.90.31.15:4717 , piece_id=5IVMKFZ3DHM3VWRV4O7BNPOIMLNWNSZDI3F4MALJSSEFYIRQDNMA, seg_pos={0 0}, upload=685442]: Long tail cancellation due to optimal threshold
Jan 7 07:03:41.673781307 (783h34m41.544864342s) [noise=true , address=145.79.227.36:4520 , piece_id=XGREKISAZVBCEOENVXW42EL2JLUSRPPP6SB5KUQ2HINPUBLKWEDQ, seg_pos={0 0}, upload=685442]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12eNRsboM5zFiyBVPLuQfgb5QSAc5bZFizK9KJHudNd9nstVGyy): protocol: expected piece hash; context canceled; EOF
Jan 7 07:03:41.673788621 (783h34m41.544871656s) [noise=true , address=145.79.227.36:4520 , piece_id=XGREKISAZVBCEOENVXW42EL2JLUSRPPP6SB5KUQ2HINPUBLKWEDQ, seg_pos={0 0}, upload=685442]: Long tail cancellation due to optimal threshold
Jan 7 07:03:41.673962201 (783h34m41.545045236s) [seg_pos={0 0}, upload=685442]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Jan 7 07:03:41.674065939 (783h34m41.545148984s) [upload=685442]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 7 07:03:41.674074154 (783h34m41.545157189s) [upload=685442]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 7 07:03:41.962488256 (783h34m41.833571301s) [upload=685442]: Done uploading segments...
Jan 7 07:03:41.962515007 (783h34m41.833598043s) [upload=685442]: Upload finished. err: <nil>
Duration: 47.523548587s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
14 out of 60 is failed (threshold=3) [............X.XXX.X.X..X....X.X.....X....X.X.....X.......X..] duration:
25.655507377s trace:
https://jaeger.dev.storj.io/trace/ae99c37afb2c1b6
FAIL (2026-01-07 07:10:29 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 25.65555154s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
49 out of 60 is failed (threshold=6) [XXX.XXXXX.XXXXXX.XXXXXXXXXXX.XXXX.XXXX..XXXXX.XXXXX..XXXXX.X] duration:
5.000929168s trace:
https://jaeger.dev.storj.io/trace/7eeeadd9c6be61d8
FAIL (2026-01-07 07:10:27 UTC)
orbiter warning: context deadline exceeded
Duration: 5.000955319s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [..X....X..............X...............X...................X.] duration:
15.34623531s trace:
https://jaeger.dev.storj.io/trace/36161e151ba11c34
FAIL (2026-01-07 07:10:22 UTC)
orbiter warning: context deadline exceeded
Duration: 15.346249998s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
16 out of 60 is failed (threshold=3) [X....X..XX.X...X....X.........X..X.X..X.......XX.X....X....X] duration:
30.003202357s trace:
https://jaeger.dev.storj.io/trace/749d6cb0a3faca9
FAIL (2026-01-07 07:10:19 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: VBP6HKFPAOSXXNGJ3ZFH2FZEH7CEDXVUMVZTV4CU6SEKSAJI2Y4A) context deadline exceeded; piecestore: (Node ID: 1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3, Piece ID: MFIZRLFHH7IF6HZQNBGWNRN23HZTPEUBKHOLPLVLNLDAWBF6WWUA) context deadline exceeded
======LOGS======
Jan 7 07:09:49.912253317 (783h40m49.783336352s) [upload=685531]: Starting upload
Jan 7 07:09:49.912311719 (783h40m49.783394754s) [upload=685531]: Done starting upload
Jan 7 07:09:49.912397562 (783h40m49.783480607s) [upload=685531]: Uploading segments...
Jan 7 07:09:49.912499045 (783h40m49.78358208s) [upload=685531]: Got next segment. Inline: false
Jan 7 07:09:49.912529593 (783h40m49.783612628s) [upload=685531]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 7 07:09:49.912534292 (783h40m49.783617327s) [upload=685531]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 7 07:09:50.154836601 (783h40m50.025919646s) [seg_pos={0 0}, upload=685531]: Begin upload segment...
Jan 7 07:09:50.154884772 (783h40m50.025967807s) [seg_pos={0 0}, upload=685531]: Done begin upload segment.
Jan 7 07:09:50.154911964 (783h40m50.025994999s) [noise=true , address=45.91.116.12:4228 , piece_id=4N5X4FXT5KJSRT3WHYIDAWWKV5RIKCJPWPTIFTKDPKJSUOZWKNOA, seg_pos={0 0}, upload=685531]: Uploading piece...
Jan 7 07:09:50.155023456 (783h40m50.026106501s) [noise=true , address=145.79.227.32:4836 , piece_id=HGC56UDF3K2HQNVKZYRNQQ26M46XSFR7FTDU4R7JUW6J2NKYVSRQ, seg_pos={0 0}, upload=685531]: Uploading piece...
Jan 7 07:09:50.155710585 (783h40m50.02679363s) [noise=true , address=145.79.227.11:4730 , piece_id=KD5YMB6RAH6NSC5G6JZELW4X4JW2KKXDE7TQZ7O2L2DMVPNBJK2Q, seg_pos={0 0}, upload=685531]: Uploading piece...
Jan 7 07:09:50.156037197 (783h40m50.027120232s) [noise=true , address=145.79.227.50:4373 , piece_id=W2HMNBVJSMIFE5SNMRVMPCWC7U3NSLRAFBPGAEXZINRVBEFDY37A, seg_pos={0 0}, upload=685531]: Uploading piece...
Jan 7 07:09:50.156422019 (783h40m50.027505064s) [noise=true , address=145.79.227.15:4873 , piece_id=4CJYVMDUA47WWBPJDABKJM3SMNEM3HMCSX7BGGRRPZZPH5GNNZNA, seg_pos={0 0}, upload=685531]: Uploading piece...
Jan 7 07:09:50.156787466 (783h40m50.027870501s) [noise=true , address=145.79.227.26:4654 , piece_id=SFFZ4GJG7YRBVZT6Q6VGDWQGOWS6EM5YZEZR5PEP6SAH4267IU2Q, seg_pos={0 0}, upload=685531]: Uploading piece...
Jan 7 07:09:50.157157029 (783h40m50.028240074s) [noise=true , address=45.91.116.13:4154 , piece_id=SBKGEIEI6WWKGGWIRZ64HBVOAQFW573MRVCL6MQSOZZ326KVCKGQ, seg_pos={0 0}, upload=685531]: Uploading piece...
Jan 7 07:09:50.157625592 (783h40m50.028708637s) [noise=true , address=145.79.227.5:4221 , piece_id=OLX6ELYP3YMMAVY5IH6MUI6WYLMF7AP2GA4UZ4S43R6XXSHLQAWA, seg_pos={0 0}, upload=685531]: Uploading piece...
Jan 7 07:09:50.158042856 (783h40m50.029125901s) [noise=true , address=77.90.31.13:4629 , piece_id=VBP6HKFPAOSXXNGJ3ZFH2FZEH7CEDXVUMVZTV4CU6SEKSAJI2Y4A, seg_pos={0 0}, upload=685531]: Uploading piece...
Jan 7 07:09:50.158475980 (783h40m50.029559015s) [noise=true , address=145.79.227.4:4185 , piece_id=DGB7ACJN5EI3IX3K6OALFU6EILFCPVXLFNMS2ITRU37UAPBO4NNA, seg_pos={0 0}, upload=685531]: Uploading piece...
Jan 7 07:09:50.158799426 (783h40m50.029882471s) [noise=true , address=145.79.227.19:4079 , piece_id=MZ7IBFC4ESSIB2DTHVRGHY2S4U7CN6G6EYUD3O62X4QRWY53EVXA, seg_pos={0 0}, upload=685531]: Uploading piece...
Jan 7 07:09:50.159313956 (783h40m50.030397001s) [noise=true , address=145.79.227.33:4869 , piece_id=6X2FVD2LO3A626BOVLE34HJDB6DXDXYUGBHXMYOQU44MNIU3TSJQ, seg_pos={0 0}, upload=685531]: Uploading piece...
Jan 7 07:09:50.160224019 (783h40m50.031307054s) [noise=true , address=145.79.227.25:4619 , piece_id=DTCUYRHXYTETSC42E6SR3EW435QJ5SIXI4ATZLHPPMHYNRM4PHSA, seg_pos={0 0}, upload=685531]: Uploading piece...
Jan 7 07:09:50.160695307 (783h40m50.031778342s) [noise=true , address=145.79.227.10:4401 , piece_id=NMBDEHVIELIFNSXJV2D57MQWA6P7GLSKX7RPL5L6TL5JD74XQR4Q, seg_pos={0 0}, upload=685531]: Uploading piece...
Jan 7 07:09:50.161221498 (783h40m50.032304534s) [noise=true , address=77.90.31.19:4842 , piece_id=MFIZRLFHH7IF6HZQNBGWNRN23HZTPEUBKHOLPLVLNLDAWBF6WWUA, seg_pos={0 0}, upload=685531]: Uploading piece...
Jan 7 07:09:50.163339983 (783h40m50.034423028s) [noise=true , address=145.79.227.20:4438 , piece_id=NLQC6NY7JIP3P32PTRGXLTZBLZOCJ6Q4OXQDGAZGOCK2WREXUAEQ, seg_pos={0 0}, upload=685531]: Uploading piece...
Jan 7 07:09:51.760026676 (783h40m51.631109711s) [upload=685531]: Next returned nil segment
Jan 7 07:09:51.760038599 (783h40m51.631121635s) [upload=685531]: Waiting for error group managing segments...
Jan 7 07:09:52.141967687 (783h40m52.013050722s) [noise=true , address=145.79.227.15:4873 , piece_id=4CJYVMDUA47WWBPJDABKJM3SMNEM3HMCSX7BGGRRPZZPH5GNNZNA, seg_pos={0 0}, upload=685531]: Done uploading piece. err: <nil>
Jan 7 07:09:52.142268360 (783h40m52.013351395s) [noise=true , address=145.79.227.32:4836 , piece_id=HGC56UDF3K2HQNVKZYRNQQ26M46XSFR7FTDU4R7JUW6J2NKYVSRQ, seg_pos={0 0}, upload=685531]: Done uploading piece. err: <nil>
Jan 7 07:09:52.144581464 (783h40m52.015664499s) [noise=true , address=145.79.227.10:4401 , piece_id=NMBDEHVIELIFNSXJV2D57MQWA6P7GLSKX7RPL5L6TL5JD74XQR4Q, seg_pos={0 0}, upload=685531]: Done uploading piece. err: <nil>
Jan 7 07:09:52.147295201 (783h40m52.018378236s) [noise=true , address=145.79.227.4:4185 , piece_id=DGB7ACJN5EI3IX3K6OALFU6EILFCPVXLFNMS2ITRU37UAPBO4NNA, seg_pos={0 0}, upload=685531]: Done uploading piece. err: <nil>
Jan 7 07:09:52.147955258 (783h40m52.019038293s) [noise=true , address=145.79.227.26:4654 , piece_id=SFFZ4GJG7YRBVZT6Q6VGDWQGOWS6EM5YZEZR5PEP6SAH4267IU2Q, seg_pos={0 0}, upload=685531]: Done uploading piece. err: <nil>
Jan 7 07:09:52.150948688 (783h40m52.022031733s) [noise=true , address=145.79.227.20:4438 , piece_id=NLQC6NY7JIP3P32PTRGXLTZBLZOCJ6Q4OXQDGAZGOCK2WREXUAEQ, seg_pos={0 0}, upload=685531]: Done uploading piece. err: <nil>
Jan 7 07:09:52.155810747 (783h40m52.026893782s) [noise=true , address=145.79.227.5:4221 , piece_id=OLX6ELYP3YMMAVY5IH6MUI6WYLMF7AP2GA4UZ4S43R6XXSHLQAWA, seg_pos={0 0}, upload=685531]: Done uploading piece. err: <nil>
Jan 7 07:09:52.166232551 (783h40m52.037315586s) [noise=true , address=145.79.227.50:4373 , piece_id=W2HMNBVJSMIFE5SNMRVMPCWC7U3NSLRAFBPGAEXZINRVBEFDY37A, seg_pos={0 0}, upload=685531]: Done uploading piece. err: <nil>
Jan 7 07:09:52.167197850 (783h40m52.038280885s) [noise=true , address=145.79.227.11:4730 , piece_id=KD5YMB6RAH6NSC5G6JZELW4X4JW2KKXDE7TQZ7O2L2DMVPNBJK2Q, seg_pos={0 0}, upload=685531]: Done uploading piece. err: <nil>
Jan 7 07:09:52.168378026 (783h40m52.039461071s) [noise=true , address=145.79.227.19:4079 , piece_id=MZ7IBFC4ESSIB2DTHVRGHY2S4U7CN6G6EYUD3O62X4QRWY53EVXA, seg_pos={0 0}, upload=685531]: Done uploading piece. err: <nil>
Jan 7 07:09:52.219682267 (783h40m52.090765302s) [noise=true , address=145.79.227.25:4619 , piece_id=DTCUYRHXYTETSC42E6SR3EW435QJ5SIXI4ATZLHPPMHYNRM4PHSA, seg_pos={0 0}, upload=685531]: Done uploading piece. err: <nil>
Jan 7 07:09:52.522922412 (783h40m52.394005457s) [noise=true , address=77.90.31.19:4842 , piece_id=MFIZRLFHH7IF6HZQNBGWNRN23HZTPEUBKHOLPLVLNLDAWBF6WWUA, seg_pos={0 0}, upload=685531]: Done uploading piece. err: <nil>
Jan 7 07:09:52.557626314 (783h40m52.428709349s) [noise=true , address=77.90.31.13:4629 , piece_id=VBP6HKFPAOSXXNGJ3ZFH2FZEH7CEDXVUMVZTV4CU6SEKSAJI2Y4A, seg_pos={0 0}, upload=685531]: Done uploading piece. err: <nil>
Jan 7 07:09:52.557649277 (783h40m52.428732322s) [seg_pos={0 0}, upload=685531]: Segment reached optimal threshold of 13 pieces.
Jan 7 07:09:52.558035883 (783h40m52.429118918s) [noise=true , address=45.91.116.13:4154 , piece_id=SBKGEIEI6WWKGGWIRZ64HBVOAQFW573MRVCL6MQSOZZ326KVCKGQ, seg_pos={0 0}, upload=685531]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF
Jan 7 07:09:52.558046694 (783h40m52.42912974s) [noise=true , address=45.91.116.13:4154 , piece_id=SBKGEIEI6WWKGGWIRZ64HBVOAQFW573MRVCL6MQSOZZ326KVCKGQ, seg_pos={0 0}, upload=685531]: Long tail cancellation due to optimal threshold
Jan 7 07:09:52.558180408 (783h40m52.429263443s) [noise=true , address=145.79.227.33:4869 , piece_id=6X2FVD2LO3A626BOVLE34HJDB6DXDXYUGBHXMYOQU44MNIU3TSJQ, seg_pos={0 0}, upload=685531]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF
Jan 7 07:09:52.558189035 (783h40m52.429272081s) [noise=true , address=145.79.227.33:4869 , piece_id=6X2FVD2LO3A626BOVLE34HJDB6DXDXYUGBHXMYOQU44MNIU3TSJQ, seg_pos={0 0}, upload=685531]: Long tail cancellation due to optimal threshold
Jan 7 07:09:52.558263677 (783h40m52.429346753s) [noise=true , address=45.91.116.12:4228 , piece_id=4N5X4FXT5KJSRT3WHYIDAWWKV5RIKCJPWPTIFTKDPKJSUOZWKNOA, seg_pos={0 0}, upload=685531]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF
Jan 7 07:09:52.558272905 (783h40m52.42935594s) [noise=true , address=45.91.116.12:4228 , piece_id=4N5X4FXT5KJSRT3WHYIDAWWKV5RIKCJPWPTIFTKDPKJSUOZWKNOA, seg_pos={0 0}, upload=685531]: Long tail cancellation due to optimal threshold
Jan 7 07:09:52.558296260 (783h40m52.429379295s) [seg_pos={0 0}, upload=685531]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 7 07:09:52.558425746 (783h40m52.429508801s) [upload=685531]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 7 07:09:52.558433801 (783h40m52.429516837s) [upload=685531]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 7 07:09:52.845394557 (783h40m52.716477602s) [upload=685531]: Done uploading segments...
Jan 7 07:09:52.845415798 (783h40m52.716498833s) [upload=685531]: Upload finished. err: <nil>
Duration: 30.003404212s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
5 out of 60 is failed (threshold=3) [...X....X..............X...............X...................X] duration:
1m0.001665359s trace:
https://jaeger.dev.storj.io/trace/36161e151ba11c34
FAIL (2026-01-07 07:10:06 UTC)
orbiter warning: context deadline exceeded
Duration: 1m0.001689826s
core
uplink
us-select-1
us1-placement-20
|