Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
17 out of 60 is failed (threshold=3) [XX.X..X..X..X..XX.......X..X......X....X...X..X.X.X.....X...] duration:
5.975186724s trace:
https://jaeger.dev.storj.io/trace/32a160c1d95b9ce1
FAIL (2025-12-30 06:30:01 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: KCDCBQ3JCAYUP26TFIRYKWXRQ5S7Q7JLWHY4XDB3FXU263DMGJRA) context deadline exceeded; piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: FCPRQ6RUSDVTSZIK6QFECZXB3T6FE6SWYH7DNEPICJW7QWFQK3EQ) context deadline exceeded
======LOGS======
Dec 30 06:27:31.479172017 (590h58m31.350255062s) [upload=518086]: Starting upload
Dec 30 06:27:31.479271175 (590h58m31.35035421s) [upload=518086]: Done starting upload
Dec 30 06:27:31.479402385 (590h58m31.35048542s) [upload=518086]: Uploading segments...
Dec 30 06:27:31.480006245 (590h58m31.35108929s) [upload=518086]: Got next segment. Inline: false
Dec 30 06:27:31.480174686 (590h58m31.351257721s) [upload=518086]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 06:27:31.480182702 (590h58m31.351265737s) [upload=518086]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 06:27:31.727629458 (590h58m31.598712493s) [seg_pos={0 0}, upload=518086]: Begin upload segment...
Dec 30 06:27:31.727685714 (590h58m31.598768759s) [seg_pos={0 0}, upload=518086]: Done begin upload segment.
Dec 30 06:27:31.727714169 (590h58m31.598797214s) [noise=true , address=77.90.31.21:4880 , piece_id=KCDCBQ3JCAYUP26TFIRYKWXRQ5S7Q7JLWHY4XDB3FXU263DMGJRA, seg_pos={0 0}, upload=518086]: Uploading piece...
Dec 30 06:27:31.727728035 (590h58m31.59881108s) [noise=true , address=145.79.227.17:4972 , piece_id=3Q2WOCLVLN4546C4TTIHVDTEI2A3NXCAKTUD337OUDAN6KSW7AOQ, seg_pos={0 0}, upload=518086]: Uploading piece...
Dec 30 06:27:31.727802376 (590h58m31.598885421s) [noise=true , address=145.79.227.13:4804 , piece_id=FCZ4CYT6QT3IHOUC6R3H57NOOOMPDM3Z6FQ3TPOH4MR5QNZR7STQ, seg_pos={0 0}, upload=518086]: Uploading piece...
Dec 30 06:27:31.729155453 (590h58m31.600238488s) [noise=true , address=145.79.227.31:4799 , piece_id=SAIJ2QPHM5AGSRV62E7TD2A4YBETORXUCK5YUK4EZQUKN3TUF64Q, seg_pos={0 0}, upload=518086]: Uploading piece...
Dec 30 06:27:31.729401191 (590h58m31.600484236s) [noise=true , address=145.79.227.22:4509 , piece_id=7U75JX2T56P7KP3CJIOI24XXY3ED4VAEIMQNJSPHKS5W7YF3FXCA, seg_pos={0 0}, upload=518086]: Uploading piece...
Dec 30 06:27:31.729850997 (590h58m31.600934032s) [noise=true , address=77.90.31.17:4773 , piece_id=NGPVC3N4UPXTO42JJIHGCQMMZU4MS3C36A3KZIYSGOF6B3PX4EJQ, seg_pos={0 0}, upload=518086]: Uploading piece...
Dec 30 06:27:31.730681368 (590h58m31.601764404s) [noise=true , address=145.79.227.45:4230 , piece_id=G64X3YOMTVLAIWQY5RIFY6K5EHDSUIZWCN2RDWK6LRVN3XUZBLKA, seg_pos={0 0}, upload=518086]: Uploading piece...
Dec 30 06:27:31.731315676 (590h58m31.602398721s) [noise=true , address=145.79.227.27:4689 , piece_id=C2RDOQOUDRFIHSAPQIJNYFHZJFQUQWPLOKJUWLPNFMKCBA5H4XLA, seg_pos={0 0}, upload=518086]: Uploading piece...
Dec 30 06:27:31.732192586 (590h58m31.603275631s) [noise=true , address=77.90.31.12:4590 , piece_id=FCPRQ6RUSDVTSZIK6QFECZXB3T6FE6SWYH7DNEPICJW7QWFQK3EQ, seg_pos={0 0}, upload=518086]: Uploading piece...
Dec 30 06:27:31.732619839 (590h58m31.603702884s) [noise=true , address=45.91.116.5:4298 , piece_id=XSQHRQH6UQGLUDFGDHXPNVJSTTRBSBDIEELEH4JC5GEEQ2M4KQ5A, seg_pos={0 0}, upload=518086]: Uploading piece...
Dec 30 06:27:31.733515114 (590h58m31.604598159s) [noise=true , address=77.90.31.13:4626 , piece_id=FWF4DQ4MPPWF2SKBBZLNGV7KWQ23BEA7UIS6FHDKINPB6YRAZARQ, seg_pos={0 0}, upload=518086]: Uploading piece...
Dec 30 06:27:31.734041075 (590h58m31.60512412s) [noise=true , address=145.79.227.20:4438 , piece_id=A623O6EK2ZQB6P37KJG6L7JYZICD5UY6JOYVKZ2BN2SMBUOMGMEA, seg_pos={0 0}, upload=518086]: Uploading piece...
Dec 30 06:27:31.734770875 (590h58m31.60585392s) [noise=true , address=145.79.227.15:4873 , piece_id=33B56NW26Q23FA3A4AZ74ZZJYAAZ5M4ZSBDX5JE3M2BX4RVUI2YQ, seg_pos={0 0}, upload=518086]: Uploading piece...
Dec 30 06:27:31.735257321 (590h58m31.606340366s) [noise=true , address=145.79.227.52:4445 , piece_id=Z4HTRQZLIIBDE34RDPNDZR5BGASGJRJI4Q6DCEW4T3CHMI2PREBA, seg_pos={0 0}, upload=518086]: Uploading piece...
Dec 30 06:27:31.736066582 (590h58m31.607149617s) [noise=true , address=145.79.227.7:4293 , piece_id=2WGGEODE53DFZXSKPINQLR7G6AMBEH7C3IEHX44WCRHWOTXWQT5A, seg_pos={0 0}, upload=518086]: Uploading piece...
Dec 30 06:27:31.737627635 (590h58m31.60871067s) [noise=true , address=145.79.227.49:4339 , piece_id=2FQP72I6EC5DPC5B6B6NFRMPQT7QYWVZFOZFWZFQFMVRFO4GCT2A, seg_pos={0 0}, upload=518086]: Uploading piece...
Dec 30 06:27:33.442981428 (590h58m33.314064473s) [upload=518086]: Next returned nil segment
Dec 30 06:27:33.443000564 (590h58m33.3140836s) [upload=518086]: Waiting for error group managing segments...
Dec 30 06:27:33.726010138 (590h58m33.597093173s) [noise=true , address=145.79.227.27:4689 , piece_id=C2RDOQOUDRFIHSAPQIJNYFHZJFQUQWPLOKJUWLPNFMKCBA5H4XLA, seg_pos={0 0}, upload=518086]: Done uploading piece. err: <nil>
Dec 30 06:27:33.726365194 (590h58m33.597448229s) [noise=true , address=145.79.227.45:4230 , piece_id=G64X3YOMTVLAIWQY5RIFY6K5EHDSUIZWCN2RDWK6LRVN3XUZBLKA, seg_pos={0 0}, upload=518086]: Done uploading piece. err: <nil>
Dec 30 06:27:33.731331761 (590h58m33.602414806s) [noise=true , address=145.79.227.52:4445 , piece_id=Z4HTRQZLIIBDE34RDPNDZR5BGASGJRJI4Q6DCEW4T3CHMI2PREBA, seg_pos={0 0}, upload=518086]: Done uploading piece. err: <nil>
Dec 30 06:27:33.732488724 (590h58m33.603571759s) [noise=true , address=145.79.227.13:4804 , piece_id=FCZ4CYT6QT3IHOUC6R3H57NOOOMPDM3Z6FQ3TPOH4MR5QNZR7STQ, seg_pos={0 0}, upload=518086]: Done uploading piece. err: <nil>
Dec 30 06:27:33.742934164 (590h58m33.614017199s) [noise=true , address=145.79.227.15:4873 , piece_id=33B56NW26Q23FA3A4AZ74ZZJYAAZ5M4ZSBDX5JE3M2BX4RVUI2YQ, seg_pos={0 0}, upload=518086]: Done uploading piece. err: <nil>
Dec 30 06:27:33.746966712 (590h58m33.618049747s) [noise=true , address=145.79.227.31:4799 , piece_id=SAIJ2QPHM5AGSRV62E7TD2A4YBETORXUCK5YUK4EZQUKN3TUF64Q, seg_pos={0 0}, upload=518086]: Done uploading piece. err: <nil>
Dec 30 06:27:33.781208554 (590h58m33.652291599s) [noise=true , address=145.79.227.7:4293 , piece_id=2WGGEODE53DFZXSKPINQLR7G6AMBEH7C3IEHX44WCRHWOTXWQT5A, seg_pos={0 0}, upload=518086]: Done uploading piece. err: <nil>
Dec 30 06:27:33.807290017 (590h58m33.678373052s) [noise=true , address=145.79.227.20:4438 , piece_id=A623O6EK2ZQB6P37KJG6L7JYZICD5UY6JOYVKZ2BN2SMBUOMGMEA, seg_pos={0 0}, upload=518086]: Done uploading piece. err: <nil>
Dec 30 06:27:33.809530753 (590h58m33.680613798s) [noise=true , address=145.79.227.22:4509 , piece_id=7U75JX2T56P7KP3CJIOI24XXY3ED4VAEIMQNJSPHKS5W7YF3FXCA, seg_pos={0 0}, upload=518086]: Done uploading piece. err: <nil>
Dec 30 06:27:33.812826569 (590h58m33.683909604s) [noise=true , address=145.79.227.17:4972 , piece_id=3Q2WOCLVLN4546C4TTIHVDTEI2A3NXCAKTUD337OUDAN6KSW7AOQ, seg_pos={0 0}, upload=518086]: Done uploading piece. err: <nil>
Dec 30 06:27:33.900372450 (590h58m33.771455495s) [noise=true , address=77.90.31.13:4626 , piece_id=FWF4DQ4MPPWF2SKBBZLNGV7KWQ23BEA7UIS6FHDKINPB6YRAZARQ, seg_pos={0 0}, upload=518086]: Done uploading piece. err: <nil>
Dec 30 06:27:33.916464471 (590h58m33.787547506s) [noise=true , address=77.90.31.21:4880 , piece_id=KCDCBQ3JCAYUP26TFIRYKWXRQ5S7Q7JLWHY4XDB3FXU263DMGJRA, seg_pos={0 0}, upload=518086]: Done uploading piece. err: <nil>
Dec 30 06:27:34.018888070 (590h58m33.889971115s) [noise=true , address=77.90.31.12:4590 , piece_id=FCPRQ6RUSDVTSZIK6QFECZXB3T6FE6SWYH7DNEPICJW7QWFQK3EQ, seg_pos={0 0}, upload=518086]: Done uploading piece. err: <nil>
Dec 30 06:27:34.018913669 (590h58m33.889996714s) [seg_pos={0 0}, upload=518086]: Segment reached optimal threshold of 13 pieces.
Dec 30 06:27:34.019155831 (590h58m33.890238876s) [noise=true , address=77.90.31.17:4773 , piece_id=NGPVC3N4UPXTO42JJIHGCQMMZU4MS3C36A3KZIYSGOF6B3PX4EJQ, seg_pos={0 0}, upload=518086]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: expected piece hash; context canceled; EOF
Dec 30 06:27:34.019166861 (590h58m33.890249906s) [noise=true , address=77.90.31.17:4773 , piece_id=NGPVC3N4UPXTO42JJIHGCQMMZU4MS3C36A3KZIYSGOF6B3PX4EJQ, seg_pos={0 0}, upload=518086]: Long tail cancellation due to optimal threshold
Dec 30 06:27:34.019372272 (590h58m33.890455318s) [noise=true , address=45.91.116.5:4298 , piece_id=XSQHRQH6UQGLUDFGDHXPNVJSTTRBSBDIEELEH4JC5GEEQ2M4KQ5A, seg_pos={0 0}, upload=518086]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF
Dec 30 06:27:34.019400827 (590h58m33.890483862s) [noise=true , address=45.91.116.5:4298 , piece_id=XSQHRQH6UQGLUDFGDHXPNVJSTTRBSBDIEELEH4JC5GEEQ2M4KQ5A, seg_pos={0 0}, upload=518086]: Long tail cancellation due to optimal threshold
Dec 30 06:27:34.019477783 (590h58m33.890560838s) [noise=true , address=145.79.227.49:4339 , piece_id=2FQP72I6EC5DPC5B6B6NFRMPQT7QYWVZFOZFWZFQFMVRFO4GCT2A, seg_pos={0 0}, upload=518086]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF
Dec 30 06:27:34.019487012 (590h58m33.890570057s) [noise=true , address=145.79.227.49:4339 , piece_id=2FQP72I6EC5DPC5B6B6NFRMPQT7QYWVZFOZFWZFQFMVRFO4GCT2A, seg_pos={0 0}, upload=518086]: Long tail cancellation due to optimal threshold
Dec 30 06:27:34.019511157 (590h58m33.890594192s) [seg_pos={0 0}, upload=518086]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 06:27:34.019579547 (590h58m33.890662582s) [upload=518086]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:27:34.019584216 (590h58m33.890667251s) [upload=518086]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:27:34.313480950 (590h58m34.184563985s) [upload=518086]: Done uploading segments...
Dec 30 06:27:34.313532458 (590h58m34.184615493s) [upload=518086]: Upload finished. err: <nil>
Duration: 5.975217102s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................X..................] duration:
8.667076962s trace:
https://jaeger.dev.storj.io/trace/deb96436003dee
FAIL (2025-12-30 06:29:54 UTC)
orbiter warning: read tcp 188.166.225.170:52458->45.38.24.2:443: read: connection reset by peer
Duration: 8.667108783s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................X.................] duration:
16.212288069s trace:
https://jaeger.dev.storj.io/trace/deb96436003dee
FAIL (2025-12-30 06:29:34 UTC)
orbiter warning: read tcp 188.166.225.170:52458->45.38.24.2:443: read: connection reset by peer
Duration: 16.212317105s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [.........................X...........X..............XX...X..] duration:
3.47173039s trace:
https://jaeger.dev.storj.io/trace/5d54da3b342a05d9
FAIL (2025-12-30 06:29:34 UTC)
orbiter warning: context deadline exceeded
Duration: 3.471765827s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [..........................X...X......X............X.........] duration:
31.461410262s trace:
https://jaeger.dev.storj.io/trace/45b769fc71e13448
FAIL (2025-12-30 06:29:26 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
.577697683 (590h53m11.448780728s) [noise=true , address=193.149.164.143:4704 , piece_id=4T2IJIA6WHI5LMIUP5BLJLCD3GRY3A6PXR6G5OWJWUSYZT5ELGHQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578173528 (590h53m11.449256573s) [noise=true , address=64.49.13.27:4965 , piece_id=CO4O5F2IBZAAKBZBV3HWNNOXZOGLNNDPALWMQZ3DEM3MREASVFEA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578542992 (590h53m11.449626027s) [noise=true , address=77.90.31.15:4729 , piece_id=NVMXFEILHGT622IIPYYT4FFHC2WZLHMCZM4VT5WUPHRAPH3DX6TA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578846911 (590h53m11.449929946s) [noise=true , address=145.79.227.43:4177 , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579458044 (590h53m11.450541089s) [noise=true , address=45.91.116.11:4150 , piece_id=E66IDHTJDLYLYJIOWMOIKOOWLXCUNCMBIQ7SQBAD3VTJF6VX5JLQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579891209 (590h53m11.450974245s) [noise=true , address=145.79.227.29:4945 , piece_id=SHL73IDEI54GRCWYNWHRUSJZZ4UUXREAX54DFJLTG5QPZCF5QKIQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579915535 (590h53m11.45099858s) [noise=true , address=145.79.227.30:4772 , piece_id=PVMD5DWRRPVARTSZ6O7HN6MUVTZUYWLYSNQR5GDMNKUJZGXMYPFQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580325466 (590h53m11.451408511s) [noise=true , address=77.90.31.12:4603 , piece_id=2I4L7F2WEGK5OSLODJSJ4672URBA7DUGXWZKTYB6ZNA2TQIGVXWA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580379599 (590h53m11.451462644s) [noise=true , address=77.90.31.14:4671 , piece_id=263TRISJ5ADFTX43ULSSGBAF2KMOXAPISJ56LLE53D74XOEMQ5EA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580692044 (590h53m11.451775089s) [noise=true , address=145.79.227.3:4175 , piece_id=UCV5WQNVWEAMEDBWDNXEVSQEGXPJE7KD6YSWC5A4WXYUKUCK4NLA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580857859 (590h53m11.451940894s) [noise=true , address=77.90.31.19:4860 , piece_id=U2MNFOAD7FHCDBE4FUYOROIP3P33ROJ7K75SF5Z747EFKFTF7RGA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581089210 (590h53m11.452172255s) [noise=true , address=145.79.227.44:4203 , piece_id=WHCKQ2FRLYAZE7Q6HYZNTS4EISVHZNAPDDKH34WN33ALK4MVK2EA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581351320 (590h53m11.452434355s) [noise=true , address=64.49.13.154:4928 , piece_id=SEVBZBMNGURP5TRUEBA4PHRSA6MNANVJUKINALJUWPQWBGKVYNTA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581457091 (590h53m11.452540126s) [noise=true , address=145.79.227.34:4909 , piece_id=3QSWTNR5LGUUAY3N5HJKSPL3TRQ6WO5F5OD4VTXKWHWUXUF7PVVQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581863144 (590h53m11.452946179s) [noise=true , address=145.79.227.33:4881 , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.584010443 (590h53m11.455093478s) [noise=true , address=145.79.227.40:4070 , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.585122791 (590h53m11.456205826s) [noise=true , address=45.91.116.14:4217 , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:12.983305402 (590h53m12.854388447s) [upload=517998]: Next returned nil segment
Dec 30 06:22:12.983321313 (590h53m12.854404348s) [upload=517998]: Waiting for error group managing segments...
Dec 30 06:22:13.379703053 (590h53m13.250786088s) [noise=true , address=64.49.13.27:4965 , piece_id=CO4O5F2IBZAAKBZBV3HWNNOXZOGLNNDPALWMQZ3DEM3MREASVFEA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.418055622 (590h53m13.289138657s) [noise=true , address=145.79.227.3:4175 , piece_id=UCV5WQNVWEAMEDBWDNXEVSQEGXPJE7KD6YSWC5A4WXYUKUCK4NLA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.509034160 (590h53m13.380117205s) [noise=true , address=64.49.13.154:4928 , piece_id=SEVBZBMNGURP5TRUEBA4PHRSA6MNANVJUKINALJUWPQWBGKVYNTA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.642886154 (590h53m13.513969199s) [noise=true , address=77.90.31.14:4671 , piece_id=263TRISJ5ADFTX43ULSSGBAF2KMOXAPISJ56LLE53D74XOEMQ5EA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.788145520 (590h53m13.659228555s) [noise=true , address=77.90.31.12:4603 , piece_id=2I4L7F2WEGK5OSLODJSJ4672URBA7DUGXWZKTYB6ZNA2TQIGVXWA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.858878757 (590h53m13.729961792s) [noise=true , address=145.79.227.49:4369 , piece_id=XUEXGNJJIJ4DUBDNUV7CKF6J5JYHSLZAEPVGEDGBUBY42BIP4XEQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.879550569 (590h53m13.750633614s) [noise=true , address=145.79.227.34:4909 , piece_id=3QSWTNR5LGUUAY3N5HJKSPL3TRQ6WO5F5OD4VTXKWHWUXUF7PVVQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.940897723 (590h53m13.811980769s) [noise=true , address=77.90.31.19:4860 , piece_id=U2MNFOAD7FHCDBE4FUYOROIP3P33ROJ7K75SF5Z747EFKFTF7RGA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.989791936 (590h53m13.860874981s) [noise=true , address=45.91.116.11:4150 , piece_id=E66IDHTJDLYLYJIOWMOIKOOWLXCUNCMBIQ7SQBAD3VTJF6VX5JLQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.289516718 (590h53m14.160599753s) [noise=true , address=145.79.227.44:4203 , piece_id=WHCKQ2FRLYAZE7Q6HYZNTS4EISVHZNAPDDKH34WN33ALK4MVK2EA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.338651537 (590h53m14.209734582s) [noise=true , address=77.90.31.15:4729 , piece_id=NVMXFEILHGT622IIPYYT4FFHC2WZLHMCZM4VT5WUPHRAPH3DX6TA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.344145168 (590h53m14.215228213s) [noise=true , address=145.79.227.30:4772 , piece_id=PVMD5DWRRPVARTSZ6O7HN6MUVTZUYWLYSNQR5GDMNKUJZGXMYPFQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.667232847 (590h53m14.538315892s) [noise=true , address=193.149.164.143:4704 , piece_id=4T2IJIA6WHI5LMIUP5BLJLCD3GRY3A6PXR6G5OWJWUSYZT5ELGHQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.671388990 (590h53m14.542472025s) [noise=true , address=145.79.227.29:4945 , piece_id=SHL73IDEI54GRCWYNWHRUSJZZ4UUXREAX54DFJLTG5QPZCF5QKIQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.671415190 (590h53m14.542498226s) [seg_pos={0 0}, upload=517998]: Segment reached optimal threshold of 14 pieces.
Dec 30 06:22:14.671803600 (590h53m14.542886655s) [noise=true , address=145.79.227.40:4070 , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hU1LwL2gX7rVnrNgzaBCYB37ziqf7s7aUgtF2Uc3antReQ6me): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.671814470 (590h53m14.542897525s) [noise=true , address=145.79.227.40:4070 , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672014932 (590h53m14.543097977s) [noise=true , address=145.79.227.43:4177 , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bhN3cR22QuAYihnnB3x3R97UZRWJ8sFPP8fjQcXZ7qhNZphZG): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672022637 (590h53m14.543105672s) [noise=true , address=145.79.227.43:4177 , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672088933 (590h53m14.543171988s) [noise=true , address=145.79.227.33:4881 , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672127677 (590h53m14.543210742s) [noise=true , address=145.79.227.33:4881 , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672312739 (590h53m14.543395774s) [noise=true , address=45.91.116.14:4217 , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xSXWFiq27Dd2vBDD9vVnhDHVG59UmDKn3hVAhf3oyo5goVNaE): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672319953 (590h53m14.543402988s) [noise=true , address=45.91.116.14:4217 , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672337847 (590h53m14.543420882s) [seg_pos={0 0}, upload=517998]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 06:22:14.672452947 (590h53m14.543535982s) [upload=517998]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:22:14.672458507 (590h53m14.543541542s) [upload=517998]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:22:14.981437069 (590h53m14.852520114s) [upload=517998]: Done uploading segments...
Dec 30 06:22:14.981456546 (590h53m14.852539581s) [upload=517998]: Upload finished. err: <nil>
Duration: 31.461438497s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
17 out of 60 is failed (threshold=3) [.XX.X..X..X..X..XX.......X..X......X....X...X..X.X.X.....X..] duration:
5.752458373s trace:
https://jaeger.dev.storj.io/trace/32a160c1d95b9ce1
FAIL (2025-12-30 06:29:13 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: KCDCBQ3JCAYUP26TFIRYKWXRQ5S7Q7JLWHY4XDB3FXU263DMGJRA) context deadline exceeded; piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: FCPRQ6RUSDVTSZIK6QFECZXB3T6FE6SWYH7DNEPICJW7QWFQK3EQ) context deadline exceeded
======LOGS======
Dec 30 06:27:31.479172017 (590h58m31.350255062s) [upload=518086]: Starting upload
Dec 30 06:27:31.479271175 (590h58m31.35035421s) [upload=518086]: Done starting upload
Dec 30 06:27:31.479402385 (590h58m31.35048542s) [upload=518086]: Uploading segments...
Dec 30 06:27:31.480006245 (590h58m31.35108929s) [upload=518086]: Got next segment. Inline: false
Dec 30 06:27:31.480174686 (590h58m31.351257721s) [upload=518086]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 06:27:31.480182702 (590h58m31.351265737s) [upload=518086]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 06:27:31.727629458 (590h58m31.598712493s) [seg_pos={0 0}, upload=518086]: Begin upload segment...
Dec 30 06:27:31.727685714 (590h58m31.598768759s) [seg_pos={0 0}, upload=518086]: Done begin upload segment.
Dec 30 06:27:31.727714169 (590h58m31.598797214s) [noise=true , address=77.90.31.21:4880 , piece_id=KCDCBQ3JCAYUP26TFIRYKWXRQ5S7Q7JLWHY4XDB3FXU263DMGJRA, seg_pos={0 0}, upload=518086]: Uploading piece...
Dec 30 06:27:31.727728035 (590h58m31.59881108s) [noise=true , address=145.79.227.17:4972 , piece_id=3Q2WOCLVLN4546C4TTIHVDTEI2A3NXCAKTUD337OUDAN6KSW7AOQ, seg_pos={0 0}, upload=518086]: Uploading piece...
Dec 30 06:27:31.727802376 (590h58m31.598885421s) [noise=true , address=145.79.227.13:4804 , piece_id=FCZ4CYT6QT3IHOUC6R3H57NOOOMPDM3Z6FQ3TPOH4MR5QNZR7STQ, seg_pos={0 0}, upload=518086]: Uploading piece...
Dec 30 06:27:31.729155453 (590h58m31.600238488s) [noise=true , address=145.79.227.31:4799 , piece_id=SAIJ2QPHM5AGSRV62E7TD2A4YBETORXUCK5YUK4EZQUKN3TUF64Q, seg_pos={0 0}, upload=518086]: Uploading piece...
Dec 30 06:27:31.729401191 (590h58m31.600484236s) [noise=true , address=145.79.227.22:4509 , piece_id=7U75JX2T56P7KP3CJIOI24XXY3ED4VAEIMQNJSPHKS5W7YF3FXCA, seg_pos={0 0}, upload=518086]: Uploading piece...
Dec 30 06:27:31.729850997 (590h58m31.600934032s) [noise=true , address=77.90.31.17:4773 , piece_id=NGPVC3N4UPXTO42JJIHGCQMMZU4MS3C36A3KZIYSGOF6B3PX4EJQ, seg_pos={0 0}, upload=518086]: Uploading piece...
Dec 30 06:27:31.730681368 (590h58m31.601764404s) [noise=true , address=145.79.227.45:4230 , piece_id=G64X3YOMTVLAIWQY5RIFY6K5EHDSUIZWCN2RDWK6LRVN3XUZBLKA, seg_pos={0 0}, upload=518086]: Uploading piece...
Dec 30 06:27:31.731315676 (590h58m31.602398721s) [noise=true , address=145.79.227.27:4689 , piece_id=C2RDOQOUDRFIHSAPQIJNYFHZJFQUQWPLOKJUWLPNFMKCBA5H4XLA, seg_pos={0 0}, upload=518086]: Uploading piece...
Dec 30 06:27:31.732192586 (590h58m31.603275631s) [noise=true , address=77.90.31.12:4590 , piece_id=FCPRQ6RUSDVTSZIK6QFECZXB3T6FE6SWYH7DNEPICJW7QWFQK3EQ, seg_pos={0 0}, upload=518086]: Uploading piece...
Dec 30 06:27:31.732619839 (590h58m31.603702884s) [noise=true , address=45.91.116.5:4298 , piece_id=XSQHRQH6UQGLUDFGDHXPNVJSTTRBSBDIEELEH4JC5GEEQ2M4KQ5A, seg_pos={0 0}, upload=518086]: Uploading piece...
Dec 30 06:27:31.733515114 (590h58m31.604598159s) [noise=true , address=77.90.31.13:4626 , piece_id=FWF4DQ4MPPWF2SKBBZLNGV7KWQ23BEA7UIS6FHDKINPB6YRAZARQ, seg_pos={0 0}, upload=518086]: Uploading piece...
Dec 30 06:27:31.734041075 (590h58m31.60512412s) [noise=true , address=145.79.227.20:4438 , piece_id=A623O6EK2ZQB6P37KJG6L7JYZICD5UY6JOYVKZ2BN2SMBUOMGMEA, seg_pos={0 0}, upload=518086]: Uploading piece...
Dec 30 06:27:31.734770875 (590h58m31.60585392s) [noise=true , address=145.79.227.15:4873 , piece_id=33B56NW26Q23FA3A4AZ74ZZJYAAZ5M4ZSBDX5JE3M2BX4RVUI2YQ, seg_pos={0 0}, upload=518086]: Uploading piece...
Dec 30 06:27:31.735257321 (590h58m31.606340366s) [noise=true , address=145.79.227.52:4445 , piece_id=Z4HTRQZLIIBDE34RDPNDZR5BGASGJRJI4Q6DCEW4T3CHMI2PREBA, seg_pos={0 0}, upload=518086]: Uploading piece...
Dec 30 06:27:31.736066582 (590h58m31.607149617s) [noise=true , address=145.79.227.7:4293 , piece_id=2WGGEODE53DFZXSKPINQLR7G6AMBEH7C3IEHX44WCRHWOTXWQT5A, seg_pos={0 0}, upload=518086]: Uploading piece...
Dec 30 06:27:31.737627635 (590h58m31.60871067s) [noise=true , address=145.79.227.49:4339 , piece_id=2FQP72I6EC5DPC5B6B6NFRMPQT7QYWVZFOZFWZFQFMVRFO4GCT2A, seg_pos={0 0}, upload=518086]: Uploading piece...
Dec 30 06:27:33.442981428 (590h58m33.314064473s) [upload=518086]: Next returned nil segment
Dec 30 06:27:33.443000564 (590h58m33.3140836s) [upload=518086]: Waiting for error group managing segments...
Dec 30 06:27:33.726010138 (590h58m33.597093173s) [noise=true , address=145.79.227.27:4689 , piece_id=C2RDOQOUDRFIHSAPQIJNYFHZJFQUQWPLOKJUWLPNFMKCBA5H4XLA, seg_pos={0 0}, upload=518086]: Done uploading piece. err: <nil>
Dec 30 06:27:33.726365194 (590h58m33.597448229s) [noise=true , address=145.79.227.45:4230 , piece_id=G64X3YOMTVLAIWQY5RIFY6K5EHDSUIZWCN2RDWK6LRVN3XUZBLKA, seg_pos={0 0}, upload=518086]: Done uploading piece. err: <nil>
Dec 30 06:27:33.731331761 (590h58m33.602414806s) [noise=true , address=145.79.227.52:4445 , piece_id=Z4HTRQZLIIBDE34RDPNDZR5BGASGJRJI4Q6DCEW4T3CHMI2PREBA, seg_pos={0 0}, upload=518086]: Done uploading piece. err: <nil>
Dec 30 06:27:33.732488724 (590h58m33.603571759s) [noise=true , address=145.79.227.13:4804 , piece_id=FCZ4CYT6QT3IHOUC6R3H57NOOOMPDM3Z6FQ3TPOH4MR5QNZR7STQ, seg_pos={0 0}, upload=518086]: Done uploading piece. err: <nil>
Dec 30 06:27:33.742934164 (590h58m33.614017199s) [noise=true , address=145.79.227.15:4873 , piece_id=33B56NW26Q23FA3A4AZ74ZZJYAAZ5M4ZSBDX5JE3M2BX4RVUI2YQ, seg_pos={0 0}, upload=518086]: Done uploading piece. err: <nil>
Dec 30 06:27:33.746966712 (590h58m33.618049747s) [noise=true , address=145.79.227.31:4799 , piece_id=SAIJ2QPHM5AGSRV62E7TD2A4YBETORXUCK5YUK4EZQUKN3TUF64Q, seg_pos={0 0}, upload=518086]: Done uploading piece. err: <nil>
Dec 30 06:27:33.781208554 (590h58m33.652291599s) [noise=true , address=145.79.227.7:4293 , piece_id=2WGGEODE53DFZXSKPINQLR7G6AMBEH7C3IEHX44WCRHWOTXWQT5A, seg_pos={0 0}, upload=518086]: Done uploading piece. err: <nil>
Dec 30 06:27:33.807290017 (590h58m33.678373052s) [noise=true , address=145.79.227.20:4438 , piece_id=A623O6EK2ZQB6P37KJG6L7JYZICD5UY6JOYVKZ2BN2SMBUOMGMEA, seg_pos={0 0}, upload=518086]: Done uploading piece. err: <nil>
Dec 30 06:27:33.809530753 (590h58m33.680613798s) [noise=true , address=145.79.227.22:4509 , piece_id=7U75JX2T56P7KP3CJIOI24XXY3ED4VAEIMQNJSPHKS5W7YF3FXCA, seg_pos={0 0}, upload=518086]: Done uploading piece. err: <nil>
Dec 30 06:27:33.812826569 (590h58m33.683909604s) [noise=true , address=145.79.227.17:4972 , piece_id=3Q2WOCLVLN4546C4TTIHVDTEI2A3NXCAKTUD337OUDAN6KSW7AOQ, seg_pos={0 0}, upload=518086]: Done uploading piece. err: <nil>
Dec 30 06:27:33.900372450 (590h58m33.771455495s) [noise=true , address=77.90.31.13:4626 , piece_id=FWF4DQ4MPPWF2SKBBZLNGV7KWQ23BEA7UIS6FHDKINPB6YRAZARQ, seg_pos={0 0}, upload=518086]: Done uploading piece. err: <nil>
Dec 30 06:27:33.916464471 (590h58m33.787547506s) [noise=true , address=77.90.31.21:4880 , piece_id=KCDCBQ3JCAYUP26TFIRYKWXRQ5S7Q7JLWHY4XDB3FXU263DMGJRA, seg_pos={0 0}, upload=518086]: Done uploading piece. err: <nil>
Dec 30 06:27:34.018888070 (590h58m33.889971115s) [noise=true , address=77.90.31.12:4590 , piece_id=FCPRQ6RUSDVTSZIK6QFECZXB3T6FE6SWYH7DNEPICJW7QWFQK3EQ, seg_pos={0 0}, upload=518086]: Done uploading piece. err: <nil>
Dec 30 06:27:34.018913669 (590h58m33.889996714s) [seg_pos={0 0}, upload=518086]: Segment reached optimal threshold of 13 pieces.
Dec 30 06:27:34.019155831 (590h58m33.890238876s) [noise=true , address=77.90.31.17:4773 , piece_id=NGPVC3N4UPXTO42JJIHGCQMMZU4MS3C36A3KZIYSGOF6B3PX4EJQ, seg_pos={0 0}, upload=518086]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: expected piece hash; context canceled; EOF
Dec 30 06:27:34.019166861 (590h58m33.890249906s) [noise=true , address=77.90.31.17:4773 , piece_id=NGPVC3N4UPXTO42JJIHGCQMMZU4MS3C36A3KZIYSGOF6B3PX4EJQ, seg_pos={0 0}, upload=518086]: Long tail cancellation due to optimal threshold
Dec 30 06:27:34.019372272 (590h58m33.890455318s) [noise=true , address=45.91.116.5:4298 , piece_id=XSQHRQH6UQGLUDFGDHXPNVJSTTRBSBDIEELEH4JC5GEEQ2M4KQ5A, seg_pos={0 0}, upload=518086]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF
Dec 30 06:27:34.019400827 (590h58m33.890483862s) [noise=true , address=45.91.116.5:4298 , piece_id=XSQHRQH6UQGLUDFGDHXPNVJSTTRBSBDIEELEH4JC5GEEQ2M4KQ5A, seg_pos={0 0}, upload=518086]: Long tail cancellation due to optimal threshold
Dec 30 06:27:34.019477783 (590h58m33.890560838s) [noise=true , address=145.79.227.49:4339 , piece_id=2FQP72I6EC5DPC5B6B6NFRMPQT7QYWVZFOZFWZFQFMVRFO4GCT2A, seg_pos={0 0}, upload=518086]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF
Dec 30 06:27:34.019487012 (590h58m33.890570057s) [noise=true , address=145.79.227.49:4339 , piece_id=2FQP72I6EC5DPC5B6B6NFRMPQT7QYWVZFOZFWZFQFMVRFO4GCT2A, seg_pos={0 0}, upload=518086]: Long tail cancellation due to optimal threshold
Dec 30 06:27:34.019511157 (590h58m33.890594192s) [seg_pos={0 0}, upload=518086]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 06:27:34.019579547 (590h58m33.890662582s) [upload=518086]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:27:34.019584216 (590h58m33.890667251s) [upload=518086]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:27:34.313480950 (590h58m34.184563985s) [upload=518086]: Done uploading segments...
Dec 30 06:27:34.313532458 (590h58m34.184615493s) [upload=518086]: Upload finished. err: <nil>
Duration: 5.752486749s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [..........................X...........X..............XX...X.] duration:
3.651357042s trace:
https://jaeger.dev.storj.io/trace/5d54da3b342a05d9
FAIL (2025-12-30 06:29:06 UTC)
orbiter warning: context deadline exceeded
Duration: 3.65138742s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
18 out of 60 is failed (threshold=3) [X.XX.X..X..X..X..XX.......X..X......X....X...X..X.X.X.....X.] duration:
5.754500012s trace:
https://jaeger.dev.storj.io/trace/32a160c1d95b9ce1
FAIL (2025-12-30 06:28:48 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: KCDCBQ3JCAYUP26TFIRYKWXRQ5S7Q7JLWHY4XDB3FXU263DMGJRA) context deadline exceeded; piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: FCPRQ6RUSDVTSZIK6QFECZXB3T6FE6SWYH7DNEPICJW7QWFQK3EQ) context deadline exceeded
======LOGS======
Dec 30 06:27:31.479172017 (590h58m31.350255062s) [upload=518086]: Starting upload
Dec 30 06:27:31.479271175 (590h58m31.35035421s) [upload=518086]: Done starting upload
Dec 30 06:27:31.479402385 (590h58m31.35048542s) [upload=518086]: Uploading segments...
Dec 30 06:27:31.480006245 (590h58m31.35108929s) [upload=518086]: Got next segment. Inline: false
Dec 30 06:27:31.480174686 (590h58m31.351257721s) [upload=518086]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 06:27:31.480182702 (590h58m31.351265737s) [upload=518086]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 06:27:31.727629458 (590h58m31.598712493s) [seg_pos={0 0}, upload=518086]: Begin upload segment...
Dec 30 06:27:31.727685714 (590h58m31.598768759s) [seg_pos={0 0}, upload=518086]: Done begin upload segment.
Dec 30 06:27:31.727714169 (590h58m31.598797214s) [noise=true , address=77.90.31.21:4880 , piece_id=KCDCBQ3JCAYUP26TFIRYKWXRQ5S7Q7JLWHY4XDB3FXU263DMGJRA, seg_pos={0 0}, upload=518086]: Uploading piece...
Dec 30 06:27:31.727728035 (590h58m31.59881108s) [noise=true , address=145.79.227.17:4972 , piece_id=3Q2WOCLVLN4546C4TTIHVDTEI2A3NXCAKTUD337OUDAN6KSW7AOQ, seg_pos={0 0}, upload=518086]: Uploading piece...
Dec 30 06:27:31.727802376 (590h58m31.598885421s) [noise=true , address=145.79.227.13:4804 , piece_id=FCZ4CYT6QT3IHOUC6R3H57NOOOMPDM3Z6FQ3TPOH4MR5QNZR7STQ, seg_pos={0 0}, upload=518086]: Uploading piece...
Dec 30 06:27:31.729155453 (590h58m31.600238488s) [noise=true , address=145.79.227.31:4799 , piece_id=SAIJ2QPHM5AGSRV62E7TD2A4YBETORXUCK5YUK4EZQUKN3TUF64Q, seg_pos={0 0}, upload=518086]: Uploading piece...
Dec 30 06:27:31.729401191 (590h58m31.600484236s) [noise=true , address=145.79.227.22:4509 , piece_id=7U75JX2T56P7KP3CJIOI24XXY3ED4VAEIMQNJSPHKS5W7YF3FXCA, seg_pos={0 0}, upload=518086]: Uploading piece...
Dec 30 06:27:31.729850997 (590h58m31.600934032s) [noise=true , address=77.90.31.17:4773 , piece_id=NGPVC3N4UPXTO42JJIHGCQMMZU4MS3C36A3KZIYSGOF6B3PX4EJQ, seg_pos={0 0}, upload=518086]: Uploading piece...
Dec 30 06:27:31.730681368 (590h58m31.601764404s) [noise=true , address=145.79.227.45:4230 , piece_id=G64X3YOMTVLAIWQY5RIFY6K5EHDSUIZWCN2RDWK6LRVN3XUZBLKA, seg_pos={0 0}, upload=518086]: Uploading piece...
Dec 30 06:27:31.731315676 (590h58m31.602398721s) [noise=true , address=145.79.227.27:4689 , piece_id=C2RDOQOUDRFIHSAPQIJNYFHZJFQUQWPLOKJUWLPNFMKCBA5H4XLA, seg_pos={0 0}, upload=518086]: Uploading piece...
Dec 30 06:27:31.732192586 (590h58m31.603275631s) [noise=true , address=77.90.31.12:4590 , piece_id=FCPRQ6RUSDVTSZIK6QFECZXB3T6FE6SWYH7DNEPICJW7QWFQK3EQ, seg_pos={0 0}, upload=518086]: Uploading piece...
Dec 30 06:27:31.732619839 (590h58m31.603702884s) [noise=true , address=45.91.116.5:4298 , piece_id=XSQHRQH6UQGLUDFGDHXPNVJSTTRBSBDIEELEH4JC5GEEQ2M4KQ5A, seg_pos={0 0}, upload=518086]: Uploading piece...
Dec 30 06:27:31.733515114 (590h58m31.604598159s) [noise=true , address=77.90.31.13:4626 , piece_id=FWF4DQ4MPPWF2SKBBZLNGV7KWQ23BEA7UIS6FHDKINPB6YRAZARQ, seg_pos={0 0}, upload=518086]: Uploading piece...
Dec 30 06:27:31.734041075 (590h58m31.60512412s) [noise=true , address=145.79.227.20:4438 , piece_id=A623O6EK2ZQB6P37KJG6L7JYZICD5UY6JOYVKZ2BN2SMBUOMGMEA, seg_pos={0 0}, upload=518086]: Uploading piece...
Dec 30 06:27:31.734770875 (590h58m31.60585392s) [noise=true , address=145.79.227.15:4873 , piece_id=33B56NW26Q23FA3A4AZ74ZZJYAAZ5M4ZSBDX5JE3M2BX4RVUI2YQ, seg_pos={0 0}, upload=518086]: Uploading piece...
Dec 30 06:27:31.735257321 (590h58m31.606340366s) [noise=true , address=145.79.227.52:4445 , piece_id=Z4HTRQZLIIBDE34RDPNDZR5BGASGJRJI4Q6DCEW4T3CHMI2PREBA, seg_pos={0 0}, upload=518086]: Uploading piece...
Dec 30 06:27:31.736066582 (590h58m31.607149617s) [noise=true , address=145.79.227.7:4293 , piece_id=2WGGEODE53DFZXSKPINQLR7G6AMBEH7C3IEHX44WCRHWOTXWQT5A, seg_pos={0 0}, upload=518086]: Uploading piece...
Dec 30 06:27:31.737627635 (590h58m31.60871067s) [noise=true , address=145.79.227.49:4339 , piece_id=2FQP72I6EC5DPC5B6B6NFRMPQT7QYWVZFOZFWZFQFMVRFO4GCT2A, seg_pos={0 0}, upload=518086]: Uploading piece...
Dec 30 06:27:33.442981428 (590h58m33.314064473s) [upload=518086]: Next returned nil segment
Dec 30 06:27:33.443000564 (590h58m33.3140836s) [upload=518086]: Waiting for error group managing segments...
Dec 30 06:27:33.726010138 (590h58m33.597093173s) [noise=true , address=145.79.227.27:4689 , piece_id=C2RDOQOUDRFIHSAPQIJNYFHZJFQUQWPLOKJUWLPNFMKCBA5H4XLA, seg_pos={0 0}, upload=518086]: Done uploading piece. err: <nil>
Dec 30 06:27:33.726365194 (590h58m33.597448229s) [noise=true , address=145.79.227.45:4230 , piece_id=G64X3YOMTVLAIWQY5RIFY6K5EHDSUIZWCN2RDWK6LRVN3XUZBLKA, seg_pos={0 0}, upload=518086]: Done uploading piece. err: <nil>
Dec 30 06:27:33.731331761 (590h58m33.602414806s) [noise=true , address=145.79.227.52:4445 , piece_id=Z4HTRQZLIIBDE34RDPNDZR5BGASGJRJI4Q6DCEW4T3CHMI2PREBA, seg_pos={0 0}, upload=518086]: Done uploading piece. err: <nil>
Dec 30 06:27:33.732488724 (590h58m33.603571759s) [noise=true , address=145.79.227.13:4804 , piece_id=FCZ4CYT6QT3IHOUC6R3H57NOOOMPDM3Z6FQ3TPOH4MR5QNZR7STQ, seg_pos={0 0}, upload=518086]: Done uploading piece. err: <nil>
Dec 30 06:27:33.742934164 (590h58m33.614017199s) [noise=true , address=145.79.227.15:4873 , piece_id=33B56NW26Q23FA3A4AZ74ZZJYAAZ5M4ZSBDX5JE3M2BX4RVUI2YQ, seg_pos={0 0}, upload=518086]: Done uploading piece. err: <nil>
Dec 30 06:27:33.746966712 (590h58m33.618049747s) [noise=true , address=145.79.227.31:4799 , piece_id=SAIJ2QPHM5AGSRV62E7TD2A4YBETORXUCK5YUK4EZQUKN3TUF64Q, seg_pos={0 0}, upload=518086]: Done uploading piece. err: <nil>
Dec 30 06:27:33.781208554 (590h58m33.652291599s) [noise=true , address=145.79.227.7:4293 , piece_id=2WGGEODE53DFZXSKPINQLR7G6AMBEH7C3IEHX44WCRHWOTXWQT5A, seg_pos={0 0}, upload=518086]: Done uploading piece. err: <nil>
Dec 30 06:27:33.807290017 (590h58m33.678373052s) [noise=true , address=145.79.227.20:4438 , piece_id=A623O6EK2ZQB6P37KJG6L7JYZICD5UY6JOYVKZ2BN2SMBUOMGMEA, seg_pos={0 0}, upload=518086]: Done uploading piece. err: <nil>
Dec 30 06:27:33.809530753 (590h58m33.680613798s) [noise=true , address=145.79.227.22:4509 , piece_id=7U75JX2T56P7KP3CJIOI24XXY3ED4VAEIMQNJSPHKS5W7YF3FXCA, seg_pos={0 0}, upload=518086]: Done uploading piece. err: <nil>
Dec 30 06:27:33.812826569 (590h58m33.683909604s) [noise=true , address=145.79.227.17:4972 , piece_id=3Q2WOCLVLN4546C4TTIHVDTEI2A3NXCAKTUD337OUDAN6KSW7AOQ, seg_pos={0 0}, upload=518086]: Done uploading piece. err: <nil>
Dec 30 06:27:33.900372450 (590h58m33.771455495s) [noise=true , address=77.90.31.13:4626 , piece_id=FWF4DQ4MPPWF2SKBBZLNGV7KWQ23BEA7UIS6FHDKINPB6YRAZARQ, seg_pos={0 0}, upload=518086]: Done uploading piece. err: <nil>
Dec 30 06:27:33.916464471 (590h58m33.787547506s) [noise=true , address=77.90.31.21:4880 , piece_id=KCDCBQ3JCAYUP26TFIRYKWXRQ5S7Q7JLWHY4XDB3FXU263DMGJRA, seg_pos={0 0}, upload=518086]: Done uploading piece. err: <nil>
Dec 30 06:27:34.018888070 (590h58m33.889971115s) [noise=true , address=77.90.31.12:4590 , piece_id=FCPRQ6RUSDVTSZIK6QFECZXB3T6FE6SWYH7DNEPICJW7QWFQK3EQ, seg_pos={0 0}, upload=518086]: Done uploading piece. err: <nil>
Dec 30 06:27:34.018913669 (590h58m33.889996714s) [seg_pos={0 0}, upload=518086]: Segment reached optimal threshold of 13 pieces.
Dec 30 06:27:34.019155831 (590h58m33.890238876s) [noise=true , address=77.90.31.17:4773 , piece_id=NGPVC3N4UPXTO42JJIHGCQMMZU4MS3C36A3KZIYSGOF6B3PX4EJQ, seg_pos={0 0}, upload=518086]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: expected piece hash; context canceled; EOF
Dec 30 06:27:34.019166861 (590h58m33.890249906s) [noise=true , address=77.90.31.17:4773 , piece_id=NGPVC3N4UPXTO42JJIHGCQMMZU4MS3C36A3KZIYSGOF6B3PX4EJQ, seg_pos={0 0}, upload=518086]: Long tail cancellation due to optimal threshold
Dec 30 06:27:34.019372272 (590h58m33.890455318s) [noise=true , address=45.91.116.5:4298 , piece_id=XSQHRQH6UQGLUDFGDHXPNVJSTTRBSBDIEELEH4JC5GEEQ2M4KQ5A, seg_pos={0 0}, upload=518086]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF
Dec 30 06:27:34.019400827 (590h58m33.890483862s) [noise=true , address=45.91.116.5:4298 , piece_id=XSQHRQH6UQGLUDFGDHXPNVJSTTRBSBDIEELEH4JC5GEEQ2M4KQ5A, seg_pos={0 0}, upload=518086]: Long tail cancellation due to optimal threshold
Dec 30 06:27:34.019477783 (590h58m33.890560838s) [noise=true , address=145.79.227.49:4339 , piece_id=2FQP72I6EC5DPC5B6B6NFRMPQT7QYWVZFOZFWZFQFMVRFO4GCT2A, seg_pos={0 0}, upload=518086]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF
Dec 30 06:27:34.019487012 (590h58m33.890570057s) [noise=true , address=145.79.227.49:4339 , piece_id=2FQP72I6EC5DPC5B6B6NFRMPQT7QYWVZFOZFWZFQFMVRFO4GCT2A, seg_pos={0 0}, upload=518086]: Long tail cancellation due to optimal threshold
Dec 30 06:27:34.019511157 (590h58m33.890594192s) [seg_pos={0 0}, upload=518086]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 06:27:34.019579547 (590h58m33.890662582s) [upload=518086]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:27:34.019584216 (590h58m33.890667251s) [upload=518086]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:27:34.313480950 (590h58m34.184563985s) [upload=518086]: Done uploading segments...
Dec 30 06:27:34.313532458 (590h58m34.184615493s) [upload=518086]: Upload finished. err: <nil>
Duration: 5.754529108s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [...........................X...X......X............X........] duration:
38.38131251s trace:
https://jaeger.dev.storj.io/trace/45b769fc71e13448
FAIL (2025-12-30 06:28:32 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
.577697683 (590h53m11.448780728s) [noise=true , address=193.149.164.143:4704 , piece_id=4T2IJIA6WHI5LMIUP5BLJLCD3GRY3A6PXR6G5OWJWUSYZT5ELGHQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578173528 (590h53m11.449256573s) [noise=true , address=64.49.13.27:4965 , piece_id=CO4O5F2IBZAAKBZBV3HWNNOXZOGLNNDPALWMQZ3DEM3MREASVFEA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578542992 (590h53m11.449626027s) [noise=true , address=77.90.31.15:4729 , piece_id=NVMXFEILHGT622IIPYYT4FFHC2WZLHMCZM4VT5WUPHRAPH3DX6TA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578846911 (590h53m11.449929946s) [noise=true , address=145.79.227.43:4177 , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579458044 (590h53m11.450541089s) [noise=true , address=45.91.116.11:4150 , piece_id=E66IDHTJDLYLYJIOWMOIKOOWLXCUNCMBIQ7SQBAD3VTJF6VX5JLQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579891209 (590h53m11.450974245s) [noise=true , address=145.79.227.29:4945 , piece_id=SHL73IDEI54GRCWYNWHRUSJZZ4UUXREAX54DFJLTG5QPZCF5QKIQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579915535 (590h53m11.45099858s) [noise=true , address=145.79.227.30:4772 , piece_id=PVMD5DWRRPVARTSZ6O7HN6MUVTZUYWLYSNQR5GDMNKUJZGXMYPFQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580325466 (590h53m11.451408511s) [noise=true , address=77.90.31.12:4603 , piece_id=2I4L7F2WEGK5OSLODJSJ4672URBA7DUGXWZKTYB6ZNA2TQIGVXWA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580379599 (590h53m11.451462644s) [noise=true , address=77.90.31.14:4671 , piece_id=263TRISJ5ADFTX43ULSSGBAF2KMOXAPISJ56LLE53D74XOEMQ5EA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580692044 (590h53m11.451775089s) [noise=true , address=145.79.227.3:4175 , piece_id=UCV5WQNVWEAMEDBWDNXEVSQEGXPJE7KD6YSWC5A4WXYUKUCK4NLA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580857859 (590h53m11.451940894s) [noise=true , address=77.90.31.19:4860 , piece_id=U2MNFOAD7FHCDBE4FUYOROIP3P33ROJ7K75SF5Z747EFKFTF7RGA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581089210 (590h53m11.452172255s) [noise=true , address=145.79.227.44:4203 , piece_id=WHCKQ2FRLYAZE7Q6HYZNTS4EISVHZNAPDDKH34WN33ALK4MVK2EA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581351320 (590h53m11.452434355s) [noise=true , address=64.49.13.154:4928 , piece_id=SEVBZBMNGURP5TRUEBA4PHRSA6MNANVJUKINALJUWPQWBGKVYNTA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581457091 (590h53m11.452540126s) [noise=true , address=145.79.227.34:4909 , piece_id=3QSWTNR5LGUUAY3N5HJKSPL3TRQ6WO5F5OD4VTXKWHWUXUF7PVVQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581863144 (590h53m11.452946179s) [noise=true , address=145.79.227.33:4881 , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.584010443 (590h53m11.455093478s) [noise=true , address=145.79.227.40:4070 , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.585122791 (590h53m11.456205826s) [noise=true , address=45.91.116.14:4217 , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:12.983305402 (590h53m12.854388447s) [upload=517998]: Next returned nil segment
Dec 30 06:22:12.983321313 (590h53m12.854404348s) [upload=517998]: Waiting for error group managing segments...
Dec 30 06:22:13.379703053 (590h53m13.250786088s) [noise=true , address=64.49.13.27:4965 , piece_id=CO4O5F2IBZAAKBZBV3HWNNOXZOGLNNDPALWMQZ3DEM3MREASVFEA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.418055622 (590h53m13.289138657s) [noise=true , address=145.79.227.3:4175 , piece_id=UCV5WQNVWEAMEDBWDNXEVSQEGXPJE7KD6YSWC5A4WXYUKUCK4NLA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.509034160 (590h53m13.380117205s) [noise=true , address=64.49.13.154:4928 , piece_id=SEVBZBMNGURP5TRUEBA4PHRSA6MNANVJUKINALJUWPQWBGKVYNTA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.642886154 (590h53m13.513969199s) [noise=true , address=77.90.31.14:4671 , piece_id=263TRISJ5ADFTX43ULSSGBAF2KMOXAPISJ56LLE53D74XOEMQ5EA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.788145520 (590h53m13.659228555s) [noise=true , address=77.90.31.12:4603 , piece_id=2I4L7F2WEGK5OSLODJSJ4672URBA7DUGXWZKTYB6ZNA2TQIGVXWA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.858878757 (590h53m13.729961792s) [noise=true , address=145.79.227.49:4369 , piece_id=XUEXGNJJIJ4DUBDNUV7CKF6J5JYHSLZAEPVGEDGBUBY42BIP4XEQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.879550569 (590h53m13.750633614s) [noise=true , address=145.79.227.34:4909 , piece_id=3QSWTNR5LGUUAY3N5HJKSPL3TRQ6WO5F5OD4VTXKWHWUXUF7PVVQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.940897723 (590h53m13.811980769s) [noise=true , address=77.90.31.19:4860 , piece_id=U2MNFOAD7FHCDBE4FUYOROIP3P33ROJ7K75SF5Z747EFKFTF7RGA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.989791936 (590h53m13.860874981s) [noise=true , address=45.91.116.11:4150 , piece_id=E66IDHTJDLYLYJIOWMOIKOOWLXCUNCMBIQ7SQBAD3VTJF6VX5JLQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.289516718 (590h53m14.160599753s) [noise=true , address=145.79.227.44:4203 , piece_id=WHCKQ2FRLYAZE7Q6HYZNTS4EISVHZNAPDDKH34WN33ALK4MVK2EA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.338651537 (590h53m14.209734582s) [noise=true , address=77.90.31.15:4729 , piece_id=NVMXFEILHGT622IIPYYT4FFHC2WZLHMCZM4VT5WUPHRAPH3DX6TA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.344145168 (590h53m14.215228213s) [noise=true , address=145.79.227.30:4772 , piece_id=PVMD5DWRRPVARTSZ6O7HN6MUVTZUYWLYSNQR5GDMNKUJZGXMYPFQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.667232847 (590h53m14.538315892s) [noise=true , address=193.149.164.143:4704 , piece_id=4T2IJIA6WHI5LMIUP5BLJLCD3GRY3A6PXR6G5OWJWUSYZT5ELGHQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.671388990 (590h53m14.542472025s) [noise=true , address=145.79.227.29:4945 , piece_id=SHL73IDEI54GRCWYNWHRUSJZZ4UUXREAX54DFJLTG5QPZCF5QKIQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.671415190 (590h53m14.542498226s) [seg_pos={0 0}, upload=517998]: Segment reached optimal threshold of 14 pieces.
Dec 30 06:22:14.671803600 (590h53m14.542886655s) [noise=true , address=145.79.227.40:4070 , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hU1LwL2gX7rVnrNgzaBCYB37ziqf7s7aUgtF2Uc3antReQ6me): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.671814470 (590h53m14.542897525s) [noise=true , address=145.79.227.40:4070 , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672014932 (590h53m14.543097977s) [noise=true , address=145.79.227.43:4177 , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bhN3cR22QuAYihnnB3x3R97UZRWJ8sFPP8fjQcXZ7qhNZphZG): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672022637 (590h53m14.543105672s) [noise=true , address=145.79.227.43:4177 , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672088933 (590h53m14.543171988s) [noise=true , address=145.79.227.33:4881 , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672127677 (590h53m14.543210742s) [noise=true , address=145.79.227.33:4881 , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672312739 (590h53m14.543395774s) [noise=true , address=45.91.116.14:4217 , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xSXWFiq27Dd2vBDD9vVnhDHVG59UmDKn3hVAhf3oyo5goVNaE): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672319953 (590h53m14.543402988s) [noise=true , address=45.91.116.14:4217 , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672337847 (590h53m14.543420882s) [seg_pos={0 0}, upload=517998]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 06:22:14.672452947 (590h53m14.543535982s) [upload=517998]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:22:14.672458507 (590h53m14.543541542s) [upload=517998]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:22:14.981437069 (590h53m14.852520114s) [upload=517998]: Done uploading segments...
Dec 30 06:22:14.981456546 (590h53m14.852539581s) [upload=517998]: Upload finished. err: <nil>
Duration: 38.381340344s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................X................] duration:
7.070537175s trace:
https://jaeger.dev.storj.io/trace/deb96436003dee
FAIL (2025-12-30 06:28:25 UTC)
orbiter warning: read tcp 188.166.225.170:52458->45.38.24.2:443: read: connection reset by peer
Duration: 7.070562564s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
5 out of 60 is failed (threshold=6) [...........................X...........X..............XX...X] duration:
5.001470301s trace:
https://jaeger.dev.storj.io/trace/5d54da3b342a05d9
FAIL (2025-12-30 06:28:16 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001495629s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
18 out of 60 is failed (threshold=3) [.X.XX.X..X..X..X..XX.......X..X......X....X...X..X.X.X.....X] duration:
30.002926158s trace:
https://jaeger.dev.storj.io/trace/32a160c1d95b9ce1
FAIL (2025-12-30 06:28:00 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb, Piece ID: KCDCBQ3JCAYUP26TFIRYKWXRQ5S7Q7JLWHY4XDB3FXU263DMGJRA) context deadline exceeded; piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: FCPRQ6RUSDVTSZIK6QFECZXB3T6FE6SWYH7DNEPICJW7QWFQK3EQ) context deadline exceeded
======LOGS======
Dec 30 06:27:31.479172017 (590h58m31.350255062s) [upload=518086]: Starting upload
Dec 30 06:27:31.479271175 (590h58m31.35035421s) [upload=518086]: Done starting upload
Dec 30 06:27:31.479402385 (590h58m31.35048542s) [upload=518086]: Uploading segments...
Dec 30 06:27:31.480006245 (590h58m31.35108929s) [upload=518086]: Got next segment. Inline: false
Dec 30 06:27:31.480174686 (590h58m31.351257721s) [upload=518086]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 06:27:31.480182702 (590h58m31.351265737s) [upload=518086]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 06:27:31.727629458 (590h58m31.598712493s) [seg_pos={0 0}, upload=518086]: Begin upload segment...
Dec 30 06:27:31.727685714 (590h58m31.598768759s) [seg_pos={0 0}, upload=518086]: Done begin upload segment.
Dec 30 06:27:31.727714169 (590h58m31.598797214s) [noise=true , address=77.90.31.21:4880 , piece_id=KCDCBQ3JCAYUP26TFIRYKWXRQ5S7Q7JLWHY4XDB3FXU263DMGJRA, seg_pos={0 0}, upload=518086]: Uploading piece...
Dec 30 06:27:31.727728035 (590h58m31.59881108s) [noise=true , address=145.79.227.17:4972 , piece_id=3Q2WOCLVLN4546C4TTIHVDTEI2A3NXCAKTUD337OUDAN6KSW7AOQ, seg_pos={0 0}, upload=518086]: Uploading piece...
Dec 30 06:27:31.727802376 (590h58m31.598885421s) [noise=true , address=145.79.227.13:4804 , piece_id=FCZ4CYT6QT3IHOUC6R3H57NOOOMPDM3Z6FQ3TPOH4MR5QNZR7STQ, seg_pos={0 0}, upload=518086]: Uploading piece...
Dec 30 06:27:31.729155453 (590h58m31.600238488s) [noise=true , address=145.79.227.31:4799 , piece_id=SAIJ2QPHM5AGSRV62E7TD2A4YBETORXUCK5YUK4EZQUKN3TUF64Q, seg_pos={0 0}, upload=518086]: Uploading piece...
Dec 30 06:27:31.729401191 (590h58m31.600484236s) [noise=true , address=145.79.227.22:4509 , piece_id=7U75JX2T56P7KP3CJIOI24XXY3ED4VAEIMQNJSPHKS5W7YF3FXCA, seg_pos={0 0}, upload=518086]: Uploading piece...
Dec 30 06:27:31.729850997 (590h58m31.600934032s) [noise=true , address=77.90.31.17:4773 , piece_id=NGPVC3N4UPXTO42JJIHGCQMMZU4MS3C36A3KZIYSGOF6B3PX4EJQ, seg_pos={0 0}, upload=518086]: Uploading piece...
Dec 30 06:27:31.730681368 (590h58m31.601764404s) [noise=true , address=145.79.227.45:4230 , piece_id=G64X3YOMTVLAIWQY5RIFY6K5EHDSUIZWCN2RDWK6LRVN3XUZBLKA, seg_pos={0 0}, upload=518086]: Uploading piece...
Dec 30 06:27:31.731315676 (590h58m31.602398721s) [noise=true , address=145.79.227.27:4689 , piece_id=C2RDOQOUDRFIHSAPQIJNYFHZJFQUQWPLOKJUWLPNFMKCBA5H4XLA, seg_pos={0 0}, upload=518086]: Uploading piece...
Dec 30 06:27:31.732192586 (590h58m31.603275631s) [noise=true , address=77.90.31.12:4590 , piece_id=FCPRQ6RUSDVTSZIK6QFECZXB3T6FE6SWYH7DNEPICJW7QWFQK3EQ, seg_pos={0 0}, upload=518086]: Uploading piece...
Dec 30 06:27:31.732619839 (590h58m31.603702884s) [noise=true , address=45.91.116.5:4298 , piece_id=XSQHRQH6UQGLUDFGDHXPNVJSTTRBSBDIEELEH4JC5GEEQ2M4KQ5A, seg_pos={0 0}, upload=518086]: Uploading piece...
Dec 30 06:27:31.733515114 (590h58m31.604598159s) [noise=true , address=77.90.31.13:4626 , piece_id=FWF4DQ4MPPWF2SKBBZLNGV7KWQ23BEA7UIS6FHDKINPB6YRAZARQ, seg_pos={0 0}, upload=518086]: Uploading piece...
Dec 30 06:27:31.734041075 (590h58m31.60512412s) [noise=true , address=145.79.227.20:4438 , piece_id=A623O6EK2ZQB6P37KJG6L7JYZICD5UY6JOYVKZ2BN2SMBUOMGMEA, seg_pos={0 0}, upload=518086]: Uploading piece...
Dec 30 06:27:31.734770875 (590h58m31.60585392s) [noise=true , address=145.79.227.15:4873 , piece_id=33B56NW26Q23FA3A4AZ74ZZJYAAZ5M4ZSBDX5JE3M2BX4RVUI2YQ, seg_pos={0 0}, upload=518086]: Uploading piece...
Dec 30 06:27:31.735257321 (590h58m31.606340366s) [noise=true , address=145.79.227.52:4445 , piece_id=Z4HTRQZLIIBDE34RDPNDZR5BGASGJRJI4Q6DCEW4T3CHMI2PREBA, seg_pos={0 0}, upload=518086]: Uploading piece...
Dec 30 06:27:31.736066582 (590h58m31.607149617s) [noise=true , address=145.79.227.7:4293 , piece_id=2WGGEODE53DFZXSKPINQLR7G6AMBEH7C3IEHX44WCRHWOTXWQT5A, seg_pos={0 0}, upload=518086]: Uploading piece...
Dec 30 06:27:31.737627635 (590h58m31.60871067s) [noise=true , address=145.79.227.49:4339 , piece_id=2FQP72I6EC5DPC5B6B6NFRMPQT7QYWVZFOZFWZFQFMVRFO4GCT2A, seg_pos={0 0}, upload=518086]: Uploading piece...
Dec 30 06:27:33.442981428 (590h58m33.314064473s) [upload=518086]: Next returned nil segment
Dec 30 06:27:33.443000564 (590h58m33.3140836s) [upload=518086]: Waiting for error group managing segments...
Dec 30 06:27:33.726010138 (590h58m33.597093173s) [noise=true , address=145.79.227.27:4689 , piece_id=C2RDOQOUDRFIHSAPQIJNYFHZJFQUQWPLOKJUWLPNFMKCBA5H4XLA, seg_pos={0 0}, upload=518086]: Done uploading piece. err: <nil>
Dec 30 06:27:33.726365194 (590h58m33.597448229s) [noise=true , address=145.79.227.45:4230 , piece_id=G64X3YOMTVLAIWQY5RIFY6K5EHDSUIZWCN2RDWK6LRVN3XUZBLKA, seg_pos={0 0}, upload=518086]: Done uploading piece. err: <nil>
Dec 30 06:27:33.731331761 (590h58m33.602414806s) [noise=true , address=145.79.227.52:4445 , piece_id=Z4HTRQZLIIBDE34RDPNDZR5BGASGJRJI4Q6DCEW4T3CHMI2PREBA, seg_pos={0 0}, upload=518086]: Done uploading piece. err: <nil>
Dec 30 06:27:33.732488724 (590h58m33.603571759s) [noise=true , address=145.79.227.13:4804 , piece_id=FCZ4CYT6QT3IHOUC6R3H57NOOOMPDM3Z6FQ3TPOH4MR5QNZR7STQ, seg_pos={0 0}, upload=518086]: Done uploading piece. err: <nil>
Dec 30 06:27:33.742934164 (590h58m33.614017199s) [noise=true , address=145.79.227.15:4873 , piece_id=33B56NW26Q23FA3A4AZ74ZZJYAAZ5M4ZSBDX5JE3M2BX4RVUI2YQ, seg_pos={0 0}, upload=518086]: Done uploading piece. err: <nil>
Dec 30 06:27:33.746966712 (590h58m33.618049747s) [noise=true , address=145.79.227.31:4799 , piece_id=SAIJ2QPHM5AGSRV62E7TD2A4YBETORXUCK5YUK4EZQUKN3TUF64Q, seg_pos={0 0}, upload=518086]: Done uploading piece. err: <nil>
Dec 30 06:27:33.781208554 (590h58m33.652291599s) [noise=true , address=145.79.227.7:4293 , piece_id=2WGGEODE53DFZXSKPINQLR7G6AMBEH7C3IEHX44WCRHWOTXWQT5A, seg_pos={0 0}, upload=518086]: Done uploading piece. err: <nil>
Dec 30 06:27:33.807290017 (590h58m33.678373052s) [noise=true , address=145.79.227.20:4438 , piece_id=A623O6EK2ZQB6P37KJG6L7JYZICD5UY6JOYVKZ2BN2SMBUOMGMEA, seg_pos={0 0}, upload=518086]: Done uploading piece. err: <nil>
Dec 30 06:27:33.809530753 (590h58m33.680613798s) [noise=true , address=145.79.227.22:4509 , piece_id=7U75JX2T56P7KP3CJIOI24XXY3ED4VAEIMQNJSPHKS5W7YF3FXCA, seg_pos={0 0}, upload=518086]: Done uploading piece. err: <nil>
Dec 30 06:27:33.812826569 (590h58m33.683909604s) [noise=true , address=145.79.227.17:4972 , piece_id=3Q2WOCLVLN4546C4TTIHVDTEI2A3NXCAKTUD337OUDAN6KSW7AOQ, seg_pos={0 0}, upload=518086]: Done uploading piece. err: <nil>
Dec 30 06:27:33.900372450 (590h58m33.771455495s) [noise=true , address=77.90.31.13:4626 , piece_id=FWF4DQ4MPPWF2SKBBZLNGV7KWQ23BEA7UIS6FHDKINPB6YRAZARQ, seg_pos={0 0}, upload=518086]: Done uploading piece. err: <nil>
Dec 30 06:27:33.916464471 (590h58m33.787547506s) [noise=true , address=77.90.31.21:4880 , piece_id=KCDCBQ3JCAYUP26TFIRYKWXRQ5S7Q7JLWHY4XDB3FXU263DMGJRA, seg_pos={0 0}, upload=518086]: Done uploading piece. err: <nil>
Dec 30 06:27:34.018888070 (590h58m33.889971115s) [noise=true , address=77.90.31.12:4590 , piece_id=FCPRQ6RUSDVTSZIK6QFECZXB3T6FE6SWYH7DNEPICJW7QWFQK3EQ, seg_pos={0 0}, upload=518086]: Done uploading piece. err: <nil>
Dec 30 06:27:34.018913669 (590h58m33.889996714s) [seg_pos={0 0}, upload=518086]: Segment reached optimal threshold of 13 pieces.
Dec 30 06:27:34.019155831 (590h58m33.890238876s) [noise=true , address=77.90.31.17:4773 , piece_id=NGPVC3N4UPXTO42JJIHGCQMMZU4MS3C36A3KZIYSGOF6B3PX4EJQ, seg_pos={0 0}, upload=518086]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: expected piece hash; context canceled; EOF
Dec 30 06:27:34.019166861 (590h58m33.890249906s) [noise=true , address=77.90.31.17:4773 , piece_id=NGPVC3N4UPXTO42JJIHGCQMMZU4MS3C36A3KZIYSGOF6B3PX4EJQ, seg_pos={0 0}, upload=518086]: Long tail cancellation due to optimal threshold
Dec 30 06:27:34.019372272 (590h58m33.890455318s) [noise=true , address=45.91.116.5:4298 , piece_id=XSQHRQH6UQGLUDFGDHXPNVJSTTRBSBDIEELEH4JC5GEEQ2M4KQ5A, seg_pos={0 0}, upload=518086]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF
Dec 30 06:27:34.019400827 (590h58m33.890483862s) [noise=true , address=45.91.116.5:4298 , piece_id=XSQHRQH6UQGLUDFGDHXPNVJSTTRBSBDIEELEH4JC5GEEQ2M4KQ5A, seg_pos={0 0}, upload=518086]: Long tail cancellation due to optimal threshold
Dec 30 06:27:34.019477783 (590h58m33.890560838s) [noise=true , address=145.79.227.49:4339 , piece_id=2FQP72I6EC5DPC5B6B6NFRMPQT7QYWVZFOZFWZFQFMVRFO4GCT2A, seg_pos={0 0}, upload=518086]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124ACRNWgJYWgCVsDkXchmT47DiUcgvtZcdGo2MzyVqZkPjyz1R): protocol: expected piece hash; context canceled; EOF
Dec 30 06:27:34.019487012 (590h58m33.890570057s) [noise=true , address=145.79.227.49:4339 , piece_id=2FQP72I6EC5DPC5B6B6NFRMPQT7QYWVZFOZFWZFQFMVRFO4GCT2A, seg_pos={0 0}, upload=518086]: Long tail cancellation due to optimal threshold
Dec 30 06:27:34.019511157 (590h58m33.890594192s) [seg_pos={0 0}, upload=518086]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 06:27:34.019579547 (590h58m33.890662582s) [upload=518086]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:27:34.019584216 (590h58m33.890667251s) [upload=518086]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:27:34.313480950 (590h58m34.184563985s) [upload=518086]: Done uploading segments...
Dec 30 06:27:34.313532458 (590h58m34.184615493s) [upload=518086]: Upload finished. err: <nil>
Duration: 30.00294782s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [............................................X...............] duration:
6.139136307s trace:
https://jaeger.dev.storj.io/trace/deb96436003dee
FAIL (2025-12-30 06:27:48 UTC)
orbiter warning: read tcp 188.166.225.170:52458->45.38.24.2:443: read: connection reset by peer
Duration: 6.139161534s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [............................X...........X..............XX...] duration:
3.497094388s trace:
https://jaeger.dev.storj.io/trace/2e49128c973fa898
FAIL (2025-12-30 06:27:35 UTC)
orbiter warning: context deadline exceeded
Duration: 3.497158338s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.............................................X..............] duration:
5.994608702s trace:
https://jaeger.dev.storj.io/trace/deb96436003dee
FAIL (2025-12-30 06:27:31 UTC)
orbiter warning: read tcp 188.166.225.170:52458->45.38.24.2:443: read: connection reset by peer
Duration: 5.99463882s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [............................X...X......X............X.......] duration:
15.367717075s trace:
https://jaeger.dev.storj.io/trace/45b769fc71e13448
FAIL (2025-12-30 06:27:19 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
.577697683 (590h53m11.448780728s) [noise=true , address=193.149.164.143:4704 , piece_id=4T2IJIA6WHI5LMIUP5BLJLCD3GRY3A6PXR6G5OWJWUSYZT5ELGHQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578173528 (590h53m11.449256573s) [noise=true , address=64.49.13.27:4965 , piece_id=CO4O5F2IBZAAKBZBV3HWNNOXZOGLNNDPALWMQZ3DEM3MREASVFEA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578542992 (590h53m11.449626027s) [noise=true , address=77.90.31.15:4729 , piece_id=NVMXFEILHGT622IIPYYT4FFHC2WZLHMCZM4VT5WUPHRAPH3DX6TA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578846911 (590h53m11.449929946s) [noise=true , address=145.79.227.43:4177 , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579458044 (590h53m11.450541089s) [noise=true , address=45.91.116.11:4150 , piece_id=E66IDHTJDLYLYJIOWMOIKOOWLXCUNCMBIQ7SQBAD3VTJF6VX5JLQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579891209 (590h53m11.450974245s) [noise=true , address=145.79.227.29:4945 , piece_id=SHL73IDEI54GRCWYNWHRUSJZZ4UUXREAX54DFJLTG5QPZCF5QKIQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579915535 (590h53m11.45099858s) [noise=true , address=145.79.227.30:4772 , piece_id=PVMD5DWRRPVARTSZ6O7HN6MUVTZUYWLYSNQR5GDMNKUJZGXMYPFQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580325466 (590h53m11.451408511s) [noise=true , address=77.90.31.12:4603 , piece_id=2I4L7F2WEGK5OSLODJSJ4672URBA7DUGXWZKTYB6ZNA2TQIGVXWA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580379599 (590h53m11.451462644s) [noise=true , address=77.90.31.14:4671 , piece_id=263TRISJ5ADFTX43ULSSGBAF2KMOXAPISJ56LLE53D74XOEMQ5EA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580692044 (590h53m11.451775089s) [noise=true , address=145.79.227.3:4175 , piece_id=UCV5WQNVWEAMEDBWDNXEVSQEGXPJE7KD6YSWC5A4WXYUKUCK4NLA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580857859 (590h53m11.451940894s) [noise=true , address=77.90.31.19:4860 , piece_id=U2MNFOAD7FHCDBE4FUYOROIP3P33ROJ7K75SF5Z747EFKFTF7RGA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581089210 (590h53m11.452172255s) [noise=true , address=145.79.227.44:4203 , piece_id=WHCKQ2FRLYAZE7Q6HYZNTS4EISVHZNAPDDKH34WN33ALK4MVK2EA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581351320 (590h53m11.452434355s) [noise=true , address=64.49.13.154:4928 , piece_id=SEVBZBMNGURP5TRUEBA4PHRSA6MNANVJUKINALJUWPQWBGKVYNTA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581457091 (590h53m11.452540126s) [noise=true , address=145.79.227.34:4909 , piece_id=3QSWTNR5LGUUAY3N5HJKSPL3TRQ6WO5F5OD4VTXKWHWUXUF7PVVQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581863144 (590h53m11.452946179s) [noise=true , address=145.79.227.33:4881 , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.584010443 (590h53m11.455093478s) [noise=true , address=145.79.227.40:4070 , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.585122791 (590h53m11.456205826s) [noise=true , address=45.91.116.14:4217 , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:12.983305402 (590h53m12.854388447s) [upload=517998]: Next returned nil segment
Dec 30 06:22:12.983321313 (590h53m12.854404348s) [upload=517998]: Waiting for error group managing segments...
Dec 30 06:22:13.379703053 (590h53m13.250786088s) [noise=true , address=64.49.13.27:4965 , piece_id=CO4O5F2IBZAAKBZBV3HWNNOXZOGLNNDPALWMQZ3DEM3MREASVFEA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.418055622 (590h53m13.289138657s) [noise=true , address=145.79.227.3:4175 , piece_id=UCV5WQNVWEAMEDBWDNXEVSQEGXPJE7KD6YSWC5A4WXYUKUCK4NLA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.509034160 (590h53m13.380117205s) [noise=true , address=64.49.13.154:4928 , piece_id=SEVBZBMNGURP5TRUEBA4PHRSA6MNANVJUKINALJUWPQWBGKVYNTA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.642886154 (590h53m13.513969199s) [noise=true , address=77.90.31.14:4671 , piece_id=263TRISJ5ADFTX43ULSSGBAF2KMOXAPISJ56LLE53D74XOEMQ5EA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.788145520 (590h53m13.659228555s) [noise=true , address=77.90.31.12:4603 , piece_id=2I4L7F2WEGK5OSLODJSJ4672URBA7DUGXWZKTYB6ZNA2TQIGVXWA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.858878757 (590h53m13.729961792s) [noise=true , address=145.79.227.49:4369 , piece_id=XUEXGNJJIJ4DUBDNUV7CKF6J5JYHSLZAEPVGEDGBUBY42BIP4XEQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.879550569 (590h53m13.750633614s) [noise=true , address=145.79.227.34:4909 , piece_id=3QSWTNR5LGUUAY3N5HJKSPL3TRQ6WO5F5OD4VTXKWHWUXUF7PVVQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.940897723 (590h53m13.811980769s) [noise=true , address=77.90.31.19:4860 , piece_id=U2MNFOAD7FHCDBE4FUYOROIP3P33ROJ7K75SF5Z747EFKFTF7RGA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.989791936 (590h53m13.860874981s) [noise=true , address=45.91.116.11:4150 , piece_id=E66IDHTJDLYLYJIOWMOIKOOWLXCUNCMBIQ7SQBAD3VTJF6VX5JLQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.289516718 (590h53m14.160599753s) [noise=true , address=145.79.227.44:4203 , piece_id=WHCKQ2FRLYAZE7Q6HYZNTS4EISVHZNAPDDKH34WN33ALK4MVK2EA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.338651537 (590h53m14.209734582s) [noise=true , address=77.90.31.15:4729 , piece_id=NVMXFEILHGT622IIPYYT4FFHC2WZLHMCZM4VT5WUPHRAPH3DX6TA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.344145168 (590h53m14.215228213s) [noise=true , address=145.79.227.30:4772 , piece_id=PVMD5DWRRPVARTSZ6O7HN6MUVTZUYWLYSNQR5GDMNKUJZGXMYPFQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.667232847 (590h53m14.538315892s) [noise=true , address=193.149.164.143:4704 , piece_id=4T2IJIA6WHI5LMIUP5BLJLCD3GRY3A6PXR6G5OWJWUSYZT5ELGHQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.671388990 (590h53m14.542472025s) [noise=true , address=145.79.227.29:4945 , piece_id=SHL73IDEI54GRCWYNWHRUSJZZ4UUXREAX54DFJLTG5QPZCF5QKIQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.671415190 (590h53m14.542498226s) [seg_pos={0 0}, upload=517998]: Segment reached optimal threshold of 14 pieces.
Dec 30 06:22:14.671803600 (590h53m14.542886655s) [noise=true , address=145.79.227.40:4070 , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hU1LwL2gX7rVnrNgzaBCYB37ziqf7s7aUgtF2Uc3antReQ6me): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.671814470 (590h53m14.542897525s) [noise=true , address=145.79.227.40:4070 , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672014932 (590h53m14.543097977s) [noise=true , address=145.79.227.43:4177 , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bhN3cR22QuAYihnnB3x3R97UZRWJ8sFPP8fjQcXZ7qhNZphZG): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672022637 (590h53m14.543105672s) [noise=true , address=145.79.227.43:4177 , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672088933 (590h53m14.543171988s) [noise=true , address=145.79.227.33:4881 , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672127677 (590h53m14.543210742s) [noise=true , address=145.79.227.33:4881 , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672312739 (590h53m14.543395774s) [noise=true , address=45.91.116.14:4217 , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xSXWFiq27Dd2vBDD9vVnhDHVG59UmDKn3hVAhf3oyo5goVNaE): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672319953 (590h53m14.543402988s) [noise=true , address=45.91.116.14:4217 , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672337847 (590h53m14.543420882s) [seg_pos={0 0}, upload=517998]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 06:22:14.672452947 (590h53m14.543535982s) [upload=517998]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:22:14.672458507 (590h53m14.543541542s) [upload=517998]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:22:14.981437069 (590h53m14.852520114s) [upload=517998]: Done uploading segments...
Dec 30 06:22:14.981456546 (590h53m14.852539581s) [upload=517998]: Upload finished. err: <nil>
Duration: 15.367744707s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
17 out of 60 is failed (threshold=3) [..X.XX.X..X..X..X..XX.......X..X......X....X...X..X.X.X.....] duration:
5.831122515s trace:
https://jaeger.dev.storj.io/trace/7fa8434019f3dc08
FAIL (2025-12-30 06:26:43 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: FN7D5LQNXZSK3XUYFXEAMDQTZ3DT5ETXGL5BJTQZG2RGFSHV2SJQ) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: N7QZZC6PLJDCPSVL3UE6DQXR4UPUGV2NK3VT533OI2CO2OO77PQQ) context deadline exceeded
======LOGS======
Dec 30 06:23:43.071179046 (590h54m42.942262101s) [upload=518023]: Starting upload
Dec 30 06:23:43.071274326 (590h54m42.942357372s) [upload=518023]: Done starting upload
Dec 30 06:23:43.071313622 (590h54m42.942396667s) [upload=518023]: Uploading segments...
Dec 30 06:23:43.071515386 (590h54m42.942598421s) [upload=518023]: Got next segment. Inline: false
Dec 30 06:23:43.071545433 (590h54m42.942628468s) [upload=518023]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 06:23:43.071551454 (590h54m42.942634499s) [upload=518023]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 06:23:43.320606664 (590h54m43.191689699s) [seg_pos={0 0}, upload=518023]: Begin upload segment...
Dec 30 06:23:43.320697006 (590h54m43.191780041s) [noise=true , address=145.79.227.12:4767 , piece_id=4QQQC5HR4WZHWHIDCZLJHF6LRUBAHREDRC55EEFDKGYSJLAI4OQQ, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.321480387 (590h54m43.192563422s) [seg_pos={0 0}, upload=518023]: Done begin upload segment.
Dec 30 06:23:43.321524281 (590h54m43.192607316s) [noise=true , address=145.79.227.27:4689 , piece_id=DBTRIDJH4JXR422HJK74ARR4HIAQ6SL3ZIPAVDHHLBY4LUFMCV3A, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.321848659 (590h54m43.192931694s) [noise=true , address=77.90.31.15:4697 , piece_id=ACAUVL7V363DYXVSTC4IUVYQQGLLWB76EUXZJCIM2A3IGEL7YYDA, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.321928801 (590h54m43.193011836s) [noise=true , address=145.79.227.38:4978 , piece_id=CSFJ52LVERSSRACZZLAW7JBKDXTSK7UFA5K23WGO5LD5GJ7DCJUA, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.322137569 (590h54m43.193220604s) [noise=true , address=45.91.116.10:4084 , piece_id=FN7D5LQNXZSK3XUYFXEAMDQTZ3DT5ETXGL5BJTQZG2RGFSHV2SJQ, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.322259802 (590h54m43.193342837s) [noise=true , address=145.79.227.15:4875 , piece_id=I7MXKFDGIF7BARQPGRBTVUU6RBPATRBZBTD6UB32KNBCQVR6ACUA, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.322421469 (590h54m43.193504514s) [noise=true , address=77.90.31.13:4627 , piece_id=SHVJ5BTIJXAC4INZYXVOX4SLVISMQ7AHGDMBNZFTWKR4674GH37A, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.322614057 (590h54m43.193697092s) [noise=true , address=145.79.227.41:4087 , piece_id=34KUIE26SDFSXQAKED4ABATH6QFFR3Z2BWGTKNFGEJWAFV3QLJYA, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.322720900 (590h54m43.193803935s) [noise=true , address=145.79.227.16:4911 , piece_id=5OTUFDE34TVGBLEL6DOAODW6MRFNUHX7IYJSFFLNK4TQ5AOIE2HQ, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.323005622 (590h54m43.194088657s) [noise=true , address=77.90.31.14:4662 , piece_id=Z3I7EEBHWDBRRKIDQAOCO5VM7LDY7LM2MMDGVSEM665H5F3TDJHA, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.323728659 (590h54m43.194811694s) [noise=true , address=77.90.31.16:4736 , piece_id=N7QZZC6PLJDCPSVL3UE6DQXR4UPUGV2NK3VT533OI2CO2OO77PQQ, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.324350754 (590h54m43.195433789s) [noise=true , address=145.79.227.9:4366 , piece_id=G2W6PSVI2KWTOVBRNOA4RJUXPVQ6HYXYR3ILZPNBHWYZH5N54FTQ, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.325895645 (590h54m43.19697868s) [noise=true , address=145.79.227.47:4265 , piece_id=IJOEH7UD7Q3LKFUEAMDDCO75VMDMFOE4OXJKQNKSESEV5DGIICBQ, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.326582463 (590h54m43.197665508s) [noise=true , address=45.91.116.9:4261 , piece_id=CQDG2JLVYT22T4NKLQLW3ZGAHDJ4CHJLFSJXZS26FLHRE46MNICQ, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.327825790 (590h54m43.198908825s) [noise=true , address=45.91.116.6:4973 , piece_id=FQDPNZYSZTFTMN5CDGC2TFF3GAAWSYYXNXYTM4KRCN7QSZBSGSQA, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.329280390 (590h54m43.200363435s) [noise=true , address=45.91.116.8:4047 , piece_id=R33VTNE36IROAHKNRCZXHWRRLUJGKEFFWI3GSG2B5IPECDWLG2LQ, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:45.023502686 (590h54m44.894585731s) [upload=518023]: Next returned nil segment
Dec 30 06:23:45.023528155 (590h54m44.8946112s) [upload=518023]: Waiting for error group managing segments...
Dec 30 06:23:45.309378197 (590h54m45.180461242s) [noise=true , address=145.79.227.38:4978 , piece_id=CSFJ52LVERSSRACZZLAW7JBKDXTSK7UFA5K23WGO5LD5GJ7DCJUA, seg_pos={0 0}, upload=518023]: Done uploading piece. err: <nil>
Dec 30 06:23:45.310585435 (590h54m45.18166848s) [noise=true , address=145.79.227.9:4366 , piece_id=G2W6PSVI2KWTOVBRNOA4RJUXPVQ6HYXYR3ILZPNBHWYZH5N54FTQ, seg_pos={0 0}, upload=518023]: Done uploading piece. err: <nil>
Dec 30 06:23:45.313225593 (590h54m45.184308628s) [noise=true , address=145.79.227.16:4911 , piece_id=5OTUFDE34TVGBLEL6DOAODW6MRFNUHX7IYJSFFLNK4TQ5AOIE2HQ, seg_pos={0 0}, upload=518023]: Done uploading piece. err: <nil>
Dec 30 06:23:45.313449610 (590h54m45.184532645s) [noise=true , address=145.79.227.41:4087 , piece_id=34KUIE26SDFSXQAKED4ABATH6QFFR3Z2BWGTKNFGEJWAFV3QLJYA, seg_pos={0 0}, upload=518023]: Done uploading piece. err: <nil>
Dec 30 06:23:45.328340344 (590h54m45.199423379s) [noise=true , address=145.79.227.27:4689 , piece_id=DBTRIDJH4JXR422HJK74ARR4HIAQ6SL3ZIPAVDHHLBY4LUFMCV3A, seg_pos={0 0}, upload=518023]: Done uploading piece. err: <nil>
Dec 30 06:23:45.329301494 (590h54m45.20038479s) [noise=true , address=145.79.227.12:4767 , piece_id=4QQQC5HR4WZHWHIDCZLJHF6LRUBAHREDRC55EEFDKGYSJLAI4OQQ, seg_pos={0 0}, upload=518023]: Done uploading piece. err: <nil>
Dec 30 06:23:45.331819288 (590h54m45.202902323s) [noise=true , address=145.79.227.15:4875 , piece_id=I7MXKFDGIF7BARQPGRBTVUU6RBPATRBZBTD6UB32KNBCQVR6ACUA, seg_pos={0 0}, upload=518023]: Done uploading piece. err: <nil>
Dec 30 06:23:45.348197735 (590h54m45.21928077s) [noise=true , address=145.79.227.47:4265 , piece_id=IJOEH7UD7Q3LKFUEAMDDCO75VMDMFOE4OXJKQNKSESEV5DGIICBQ, seg_pos={0 0}, upload=518023]: Done uploading piece. err: <nil>
Dec 30 06:23:45.583085838 (590h54m45.454168873s) [noise=true , address=77.90.31.15:4697 , piece_id=ACAUVL7V363DYXVSTC4IUVYQQGLLWB76EUXZJCIM2A3IGEL7YYDA, seg_pos={0 0}, upload=518023]: Done uploading piece. err: <nil>
Dec 30 06:23:45.634110076 (590h54m45.505193111s) [noise=true , address=45.91.116.10:4084 , piece_id=FN7D5LQNXZSK3XUYFXEAMDQTZ3DT5ETXGL5BJTQZG2RGFSHV2SJQ, seg_pos={0 0}, upload=518023]: Done uploading piece. err: <nil>
Dec 30 06:23:46.004722348 (590h54m45.875805383s) [noise=true , address=45.91.116.8:4047 , piece_id=R33VTNE36IROAHKNRCZXHWRRLUJGKEFFWI3GSG2B5IPECDWLG2LQ, seg_pos={0 0}, upload=518023]: Done uploading piece. err: <nil>
Dec 30 06:23:46.236932912 (590h54m46.108015957s) [noise=true , address=77.90.31.16:4736 , piece_id=N7QZZC6PLJDCPSVL3UE6DQXR4UPUGV2NK3VT533OI2CO2OO77PQQ, seg_pos={0 0}, upload=518023]: Done uploading piece. err: <nil>
Dec 30 06:23:46.325242185 (590h54m46.19632522s) [noise=true , address=77.90.31.14:4662 , piece_id=Z3I7EEBHWDBRRKIDQAOCO5VM7LDY7LM2MMDGVSEM665H5F3TDJHA, seg_pos={0 0}, upload=518023]: Done uploading piece. err: <nil>
Dec 30 06:23:46.325267093 (590h54m46.196350138s) [seg_pos={0 0}, upload=518023]: Segment reached optimal threshold of 13 pieces.
Dec 30 06:23:46.325517570 (590h54m46.196600615s) [noise=true , address=45.91.116.6:4973 , piece_id=FQDPNZYSZTFTMN5CDGC2TFF3GAAWSYYXNXYTM4KRCN7QSZBSGSQA, seg_pos={0 0}, upload=518023]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF
Dec 30 06:23:46.325527078 (590h54m46.196610114s) [noise=true , address=45.91.116.6:4973 , piece_id=FQDPNZYSZTFTMN5CDGC2TFF3GAAWSYYXNXYTM4KRCN7QSZBSGSQA, seg_pos={0 0}, upload=518023]: Long tail cancellation due to optimal threshold
Dec 30 06:23:46.325646135 (590h54m46.19672919s) [noise=true , address=77.90.31.13:4627 , piece_id=SHVJ5BTIJXAC4INZYXVOX4SLVISMQ7AHGDMBNZFTWKR4674GH37A, seg_pos={0 0}, upload=518023]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:59304->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:59304->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59304->77.90.31.13:4627: use of closed network connection
Dec 30 06:23:46.325676553 (590h54m46.196759588s) [noise=true , address=77.90.31.13:4627 , piece_id=SHVJ5BTIJXAC4INZYXVOX4SLVISMQ7AHGDMBNZFTWKR4674GH37A, seg_pos={0 0}, upload=518023]: Long tail cancellation due to optimal threshold
Dec 30 06:23:46.325887665 (590h54m46.1969707s) [noise=true , address=45.91.116.9:4261 , piece_id=CQDG2JLVYT22T4NKLQLW3ZGAHDJ4CHJLFSJXZS26FLHRE46MNICQ, seg_pos={0 0}, upload=518023]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF
Dec 30 06:23:46.325895149 (590h54m46.196978184s) [noise=true , address=45.91.116.9:4261 , piece_id=CQDG2JLVYT22T4NKLQLW3ZGAHDJ4CHJLFSJXZS26FLHRE46MNICQ, seg_pos={0 0}, upload=518023]: Long tail cancellation due to optimal threshold
Dec 30 06:23:46.325926509 (590h54m46.197009544s) [seg_pos={0 0}, upload=518023]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 06:23:46.326041127 (590h54m46.197124182s) [upload=518023]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:23:46.326050726 (590h54m46.197133761s) [upload=518023]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:23:46.627273810 (590h54m46.498356845s) [upload=518023]: Done uploading segments...
Dec 30 06:23:46.627299639 (590h54m46.498382684s) [upload=518023]: Upload finished. err: <nil>
Duration: 5.831148994s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [.............................X...X......X............X......] duration:
15.388487074s trace:
https://jaeger.dev.storj.io/trace/45b769fc71e13448
FAIL (2025-12-30 06:26:38 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
.577697683 (590h53m11.448780728s) [noise=true , address=193.149.164.143:4704 , piece_id=4T2IJIA6WHI5LMIUP5BLJLCD3GRY3A6PXR6G5OWJWUSYZT5ELGHQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578173528 (590h53m11.449256573s) [noise=true , address=64.49.13.27:4965 , piece_id=CO4O5F2IBZAAKBZBV3HWNNOXZOGLNNDPALWMQZ3DEM3MREASVFEA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578542992 (590h53m11.449626027s) [noise=true , address=77.90.31.15:4729 , piece_id=NVMXFEILHGT622IIPYYT4FFHC2WZLHMCZM4VT5WUPHRAPH3DX6TA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578846911 (590h53m11.449929946s) [noise=true , address=145.79.227.43:4177 , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579458044 (590h53m11.450541089s) [noise=true , address=45.91.116.11:4150 , piece_id=E66IDHTJDLYLYJIOWMOIKOOWLXCUNCMBIQ7SQBAD3VTJF6VX5JLQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579891209 (590h53m11.450974245s) [noise=true , address=145.79.227.29:4945 , piece_id=SHL73IDEI54GRCWYNWHRUSJZZ4UUXREAX54DFJLTG5QPZCF5QKIQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579915535 (590h53m11.45099858s) [noise=true , address=145.79.227.30:4772 , piece_id=PVMD5DWRRPVARTSZ6O7HN6MUVTZUYWLYSNQR5GDMNKUJZGXMYPFQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580325466 (590h53m11.451408511s) [noise=true , address=77.90.31.12:4603 , piece_id=2I4L7F2WEGK5OSLODJSJ4672URBA7DUGXWZKTYB6ZNA2TQIGVXWA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580379599 (590h53m11.451462644s) [noise=true , address=77.90.31.14:4671 , piece_id=263TRISJ5ADFTX43ULSSGBAF2KMOXAPISJ56LLE53D74XOEMQ5EA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580692044 (590h53m11.451775089s) [noise=true , address=145.79.227.3:4175 , piece_id=UCV5WQNVWEAMEDBWDNXEVSQEGXPJE7KD6YSWC5A4WXYUKUCK4NLA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580857859 (590h53m11.451940894s) [noise=true , address=77.90.31.19:4860 , piece_id=U2MNFOAD7FHCDBE4FUYOROIP3P33ROJ7K75SF5Z747EFKFTF7RGA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581089210 (590h53m11.452172255s) [noise=true , address=145.79.227.44:4203 , piece_id=WHCKQ2FRLYAZE7Q6HYZNTS4EISVHZNAPDDKH34WN33ALK4MVK2EA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581351320 (590h53m11.452434355s) [noise=true , address=64.49.13.154:4928 , piece_id=SEVBZBMNGURP5TRUEBA4PHRSA6MNANVJUKINALJUWPQWBGKVYNTA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581457091 (590h53m11.452540126s) [noise=true , address=145.79.227.34:4909 , piece_id=3QSWTNR5LGUUAY3N5HJKSPL3TRQ6WO5F5OD4VTXKWHWUXUF7PVVQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581863144 (590h53m11.452946179s) [noise=true , address=145.79.227.33:4881 , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.584010443 (590h53m11.455093478s) [noise=true , address=145.79.227.40:4070 , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.585122791 (590h53m11.456205826s) [noise=true , address=45.91.116.14:4217 , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:12.983305402 (590h53m12.854388447s) [upload=517998]: Next returned nil segment
Dec 30 06:22:12.983321313 (590h53m12.854404348s) [upload=517998]: Waiting for error group managing segments...
Dec 30 06:22:13.379703053 (590h53m13.250786088s) [noise=true , address=64.49.13.27:4965 , piece_id=CO4O5F2IBZAAKBZBV3HWNNOXZOGLNNDPALWMQZ3DEM3MREASVFEA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.418055622 (590h53m13.289138657s) [noise=true , address=145.79.227.3:4175 , piece_id=UCV5WQNVWEAMEDBWDNXEVSQEGXPJE7KD6YSWC5A4WXYUKUCK4NLA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.509034160 (590h53m13.380117205s) [noise=true , address=64.49.13.154:4928 , piece_id=SEVBZBMNGURP5TRUEBA4PHRSA6MNANVJUKINALJUWPQWBGKVYNTA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.642886154 (590h53m13.513969199s) [noise=true , address=77.90.31.14:4671 , piece_id=263TRISJ5ADFTX43ULSSGBAF2KMOXAPISJ56LLE53D74XOEMQ5EA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.788145520 (590h53m13.659228555s) [noise=true , address=77.90.31.12:4603 , piece_id=2I4L7F2WEGK5OSLODJSJ4672URBA7DUGXWZKTYB6ZNA2TQIGVXWA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.858878757 (590h53m13.729961792s) [noise=true , address=145.79.227.49:4369 , piece_id=XUEXGNJJIJ4DUBDNUV7CKF6J5JYHSLZAEPVGEDGBUBY42BIP4XEQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.879550569 (590h53m13.750633614s) [noise=true , address=145.79.227.34:4909 , piece_id=3QSWTNR5LGUUAY3N5HJKSPL3TRQ6WO5F5OD4VTXKWHWUXUF7PVVQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.940897723 (590h53m13.811980769s) [noise=true , address=77.90.31.19:4860 , piece_id=U2MNFOAD7FHCDBE4FUYOROIP3P33ROJ7K75SF5Z747EFKFTF7RGA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.989791936 (590h53m13.860874981s) [noise=true , address=45.91.116.11:4150 , piece_id=E66IDHTJDLYLYJIOWMOIKOOWLXCUNCMBIQ7SQBAD3VTJF6VX5JLQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.289516718 (590h53m14.160599753s) [noise=true , address=145.79.227.44:4203 , piece_id=WHCKQ2FRLYAZE7Q6HYZNTS4EISVHZNAPDDKH34WN33ALK4MVK2EA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.338651537 (590h53m14.209734582s) [noise=true , address=77.90.31.15:4729 , piece_id=NVMXFEILHGT622IIPYYT4FFHC2WZLHMCZM4VT5WUPHRAPH3DX6TA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.344145168 (590h53m14.215228213s) [noise=true , address=145.79.227.30:4772 , piece_id=PVMD5DWRRPVARTSZ6O7HN6MUVTZUYWLYSNQR5GDMNKUJZGXMYPFQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.667232847 (590h53m14.538315892s) [noise=true , address=193.149.164.143:4704 , piece_id=4T2IJIA6WHI5LMIUP5BLJLCD3GRY3A6PXR6G5OWJWUSYZT5ELGHQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.671388990 (590h53m14.542472025s) [noise=true , address=145.79.227.29:4945 , piece_id=SHL73IDEI54GRCWYNWHRUSJZZ4UUXREAX54DFJLTG5QPZCF5QKIQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.671415190 (590h53m14.542498226s) [seg_pos={0 0}, upload=517998]: Segment reached optimal threshold of 14 pieces.
Dec 30 06:22:14.671803600 (590h53m14.542886655s) [noise=true , address=145.79.227.40:4070 , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hU1LwL2gX7rVnrNgzaBCYB37ziqf7s7aUgtF2Uc3antReQ6me): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.671814470 (590h53m14.542897525s) [noise=true , address=145.79.227.40:4070 , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672014932 (590h53m14.543097977s) [noise=true , address=145.79.227.43:4177 , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bhN3cR22QuAYihnnB3x3R97UZRWJ8sFPP8fjQcXZ7qhNZphZG): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672022637 (590h53m14.543105672s) [noise=true , address=145.79.227.43:4177 , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672088933 (590h53m14.543171988s) [noise=true , address=145.79.227.33:4881 , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672127677 (590h53m14.543210742s) [noise=true , address=145.79.227.33:4881 , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672312739 (590h53m14.543395774s) [noise=true , address=45.91.116.14:4217 , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xSXWFiq27Dd2vBDD9vVnhDHVG59UmDKn3hVAhf3oyo5goVNaE): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672319953 (590h53m14.543402988s) [noise=true , address=45.91.116.14:4217 , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672337847 (590h53m14.543420882s) [seg_pos={0 0}, upload=517998]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 06:22:14.672452947 (590h53m14.543535982s) [upload=517998]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:22:14.672458507 (590h53m14.543541542s) [upload=517998]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:22:14.981437069 (590h53m14.852520114s) [upload=517998]: Done uploading segments...
Dec 30 06:22:14.981456546 (590h53m14.852539581s) [upload=517998]: Upload finished. err: <nil>
Duration: 15.388502372s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [.............................X...........X..............XX..] duration:
3.418315731s trace:
https://jaeger.dev.storj.io/trace/2e49128c973fa898
FAIL (2025-12-30 06:26:23 UTC)
orbiter warning: context deadline exceeded
Duration: 3.418341329s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
17 out of 60 is failed (threshold=3) [...X.XX.X..X..X..X..XX.......X..X......X....X...X..X.X.X....] duration:
6.35934115s trace:
https://jaeger.dev.storj.io/trace/7fa8434019f3dc08
FAIL (2025-12-30 06:26:22 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: FN7D5LQNXZSK3XUYFXEAMDQTZ3DT5ETXGL5BJTQZG2RGFSHV2SJQ) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: N7QZZC6PLJDCPSVL3UE6DQXR4UPUGV2NK3VT533OI2CO2OO77PQQ) context deadline exceeded
======LOGS======
Dec 30 06:23:43.071179046 (590h54m42.942262101s) [upload=518023]: Starting upload
Dec 30 06:23:43.071274326 (590h54m42.942357372s) [upload=518023]: Done starting upload
Dec 30 06:23:43.071313622 (590h54m42.942396667s) [upload=518023]: Uploading segments...
Dec 30 06:23:43.071515386 (590h54m42.942598421s) [upload=518023]: Got next segment. Inline: false
Dec 30 06:23:43.071545433 (590h54m42.942628468s) [upload=518023]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 06:23:43.071551454 (590h54m42.942634499s) [upload=518023]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 06:23:43.320606664 (590h54m43.191689699s) [seg_pos={0 0}, upload=518023]: Begin upload segment...
Dec 30 06:23:43.320697006 (590h54m43.191780041s) [noise=true , address=145.79.227.12:4767 , piece_id=4QQQC5HR4WZHWHIDCZLJHF6LRUBAHREDRC55EEFDKGYSJLAI4OQQ, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.321480387 (590h54m43.192563422s) [seg_pos={0 0}, upload=518023]: Done begin upload segment.
Dec 30 06:23:43.321524281 (590h54m43.192607316s) [noise=true , address=145.79.227.27:4689 , piece_id=DBTRIDJH4JXR422HJK74ARR4HIAQ6SL3ZIPAVDHHLBY4LUFMCV3A, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.321848659 (590h54m43.192931694s) [noise=true , address=77.90.31.15:4697 , piece_id=ACAUVL7V363DYXVSTC4IUVYQQGLLWB76EUXZJCIM2A3IGEL7YYDA, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.321928801 (590h54m43.193011836s) [noise=true , address=145.79.227.38:4978 , piece_id=CSFJ52LVERSSRACZZLAW7JBKDXTSK7UFA5K23WGO5LD5GJ7DCJUA, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.322137569 (590h54m43.193220604s) [noise=true , address=45.91.116.10:4084 , piece_id=FN7D5LQNXZSK3XUYFXEAMDQTZ3DT5ETXGL5BJTQZG2RGFSHV2SJQ, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.322259802 (590h54m43.193342837s) [noise=true , address=145.79.227.15:4875 , piece_id=I7MXKFDGIF7BARQPGRBTVUU6RBPATRBZBTD6UB32KNBCQVR6ACUA, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.322421469 (590h54m43.193504514s) [noise=true , address=77.90.31.13:4627 , piece_id=SHVJ5BTIJXAC4INZYXVOX4SLVISMQ7AHGDMBNZFTWKR4674GH37A, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.322614057 (590h54m43.193697092s) [noise=true , address=145.79.227.41:4087 , piece_id=34KUIE26SDFSXQAKED4ABATH6QFFR3Z2BWGTKNFGEJWAFV3QLJYA, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.322720900 (590h54m43.193803935s) [noise=true , address=145.79.227.16:4911 , piece_id=5OTUFDE34TVGBLEL6DOAODW6MRFNUHX7IYJSFFLNK4TQ5AOIE2HQ, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.323005622 (590h54m43.194088657s) [noise=true , address=77.90.31.14:4662 , piece_id=Z3I7EEBHWDBRRKIDQAOCO5VM7LDY7LM2MMDGVSEM665H5F3TDJHA, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.323728659 (590h54m43.194811694s) [noise=true , address=77.90.31.16:4736 , piece_id=N7QZZC6PLJDCPSVL3UE6DQXR4UPUGV2NK3VT533OI2CO2OO77PQQ, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.324350754 (590h54m43.195433789s) [noise=true , address=145.79.227.9:4366 , piece_id=G2W6PSVI2KWTOVBRNOA4RJUXPVQ6HYXYR3ILZPNBHWYZH5N54FTQ, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.325895645 (590h54m43.19697868s) [noise=true , address=145.79.227.47:4265 , piece_id=IJOEH7UD7Q3LKFUEAMDDCO75VMDMFOE4OXJKQNKSESEV5DGIICBQ, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.326582463 (590h54m43.197665508s) [noise=true , address=45.91.116.9:4261 , piece_id=CQDG2JLVYT22T4NKLQLW3ZGAHDJ4CHJLFSJXZS26FLHRE46MNICQ, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.327825790 (590h54m43.198908825s) [noise=true , address=45.91.116.6:4973 , piece_id=FQDPNZYSZTFTMN5CDGC2TFF3GAAWSYYXNXYTM4KRCN7QSZBSGSQA, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.329280390 (590h54m43.200363435s) [noise=true , address=45.91.116.8:4047 , piece_id=R33VTNE36IROAHKNRCZXHWRRLUJGKEFFWI3GSG2B5IPECDWLG2LQ, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:45.023502686 (590h54m44.894585731s) [upload=518023]: Next returned nil segment
Dec 30 06:23:45.023528155 (590h54m44.8946112s) [upload=518023]: Waiting for error group managing segments...
Dec 30 06:23:45.309378197 (590h54m45.180461242s) [noise=true , address=145.79.227.38:4978 , piece_id=CSFJ52LVERSSRACZZLAW7JBKDXTSK7UFA5K23WGO5LD5GJ7DCJUA, seg_pos={0 0}, upload=518023]: Done uploading piece. err: <nil>
Dec 30 06:23:45.310585435 (590h54m45.18166848s) [noise=true , address=145.79.227.9:4366 , piece_id=G2W6PSVI2KWTOVBRNOA4RJUXPVQ6HYXYR3ILZPNBHWYZH5N54FTQ, seg_pos={0 0}, upload=518023]: Done uploading piece. err: <nil>
Dec 30 06:23:45.313225593 (590h54m45.184308628s) [noise=true , address=145.79.227.16:4911 , piece_id=5OTUFDE34TVGBLEL6DOAODW6MRFNUHX7IYJSFFLNK4TQ5AOIE2HQ, seg_pos={0 0}, upload=518023]: Done uploading piece. err: <nil>
Dec 30 06:23:45.313449610 (590h54m45.184532645s) [noise=true , address=145.79.227.41:4087 , piece_id=34KUIE26SDFSXQAKED4ABATH6QFFR3Z2BWGTKNFGEJWAFV3QLJYA, seg_pos={0 0}, upload=518023]: Done uploading piece. err: <nil>
Dec 30 06:23:45.328340344 (590h54m45.199423379s) [noise=true , address=145.79.227.27:4689 , piece_id=DBTRIDJH4JXR422HJK74ARR4HIAQ6SL3ZIPAVDHHLBY4LUFMCV3A, seg_pos={0 0}, upload=518023]: Done uploading piece. err: <nil>
Dec 30 06:23:45.329301494 (590h54m45.20038479s) [noise=true , address=145.79.227.12:4767 , piece_id=4QQQC5HR4WZHWHIDCZLJHF6LRUBAHREDRC55EEFDKGYSJLAI4OQQ, seg_pos={0 0}, upload=518023]: Done uploading piece. err: <nil>
Dec 30 06:23:45.331819288 (590h54m45.202902323s) [noise=true , address=145.79.227.15:4875 , piece_id=I7MXKFDGIF7BARQPGRBTVUU6RBPATRBZBTD6UB32KNBCQVR6ACUA, seg_pos={0 0}, upload=518023]: Done uploading piece. err: <nil>
Dec 30 06:23:45.348197735 (590h54m45.21928077s) [noise=true , address=145.79.227.47:4265 , piece_id=IJOEH7UD7Q3LKFUEAMDDCO75VMDMFOE4OXJKQNKSESEV5DGIICBQ, seg_pos={0 0}, upload=518023]: Done uploading piece. err: <nil>
Dec 30 06:23:45.583085838 (590h54m45.454168873s) [noise=true , address=77.90.31.15:4697 , piece_id=ACAUVL7V363DYXVSTC4IUVYQQGLLWB76EUXZJCIM2A3IGEL7YYDA, seg_pos={0 0}, upload=518023]: Done uploading piece. err: <nil>
Dec 30 06:23:45.634110076 (590h54m45.505193111s) [noise=true , address=45.91.116.10:4084 , piece_id=FN7D5LQNXZSK3XUYFXEAMDQTZ3DT5ETXGL5BJTQZG2RGFSHV2SJQ, seg_pos={0 0}, upload=518023]: Done uploading piece. err: <nil>
Dec 30 06:23:46.004722348 (590h54m45.875805383s) [noise=true , address=45.91.116.8:4047 , piece_id=R33VTNE36IROAHKNRCZXHWRRLUJGKEFFWI3GSG2B5IPECDWLG2LQ, seg_pos={0 0}, upload=518023]: Done uploading piece. err: <nil>
Dec 30 06:23:46.236932912 (590h54m46.108015957s) [noise=true , address=77.90.31.16:4736 , piece_id=N7QZZC6PLJDCPSVL3UE6DQXR4UPUGV2NK3VT533OI2CO2OO77PQQ, seg_pos={0 0}, upload=518023]: Done uploading piece. err: <nil>
Dec 30 06:23:46.325242185 (590h54m46.19632522s) [noise=true , address=77.90.31.14:4662 , piece_id=Z3I7EEBHWDBRRKIDQAOCO5VM7LDY7LM2MMDGVSEM665H5F3TDJHA, seg_pos={0 0}, upload=518023]: Done uploading piece. err: <nil>
Dec 30 06:23:46.325267093 (590h54m46.196350138s) [seg_pos={0 0}, upload=518023]: Segment reached optimal threshold of 13 pieces.
Dec 30 06:23:46.325517570 (590h54m46.196600615s) [noise=true , address=45.91.116.6:4973 , piece_id=FQDPNZYSZTFTMN5CDGC2TFF3GAAWSYYXNXYTM4KRCN7QSZBSGSQA, seg_pos={0 0}, upload=518023]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF
Dec 30 06:23:46.325527078 (590h54m46.196610114s) [noise=true , address=45.91.116.6:4973 , piece_id=FQDPNZYSZTFTMN5CDGC2TFF3GAAWSYYXNXYTM4KRCN7QSZBSGSQA, seg_pos={0 0}, upload=518023]: Long tail cancellation due to optimal threshold
Dec 30 06:23:46.325646135 (590h54m46.19672919s) [noise=true , address=77.90.31.13:4627 , piece_id=SHVJ5BTIJXAC4INZYXVOX4SLVISMQ7AHGDMBNZFTWKR4674GH37A, seg_pos={0 0}, upload=518023]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:59304->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:59304->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59304->77.90.31.13:4627: use of closed network connection
Dec 30 06:23:46.325676553 (590h54m46.196759588s) [noise=true , address=77.90.31.13:4627 , piece_id=SHVJ5BTIJXAC4INZYXVOX4SLVISMQ7AHGDMBNZFTWKR4674GH37A, seg_pos={0 0}, upload=518023]: Long tail cancellation due to optimal threshold
Dec 30 06:23:46.325887665 (590h54m46.1969707s) [noise=true , address=45.91.116.9:4261 , piece_id=CQDG2JLVYT22T4NKLQLW3ZGAHDJ4CHJLFSJXZS26FLHRE46MNICQ, seg_pos={0 0}, upload=518023]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF
Dec 30 06:23:46.325895149 (590h54m46.196978184s) [noise=true , address=45.91.116.9:4261 , piece_id=CQDG2JLVYT22T4NKLQLW3ZGAHDJ4CHJLFSJXZS26FLHRE46MNICQ, seg_pos={0 0}, upload=518023]: Long tail cancellation due to optimal threshold
Dec 30 06:23:46.325926509 (590h54m46.197009544s) [seg_pos={0 0}, upload=518023]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 06:23:46.326041127 (590h54m46.197124182s) [upload=518023]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:23:46.326050726 (590h54m46.197133761s) [upload=518023]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:23:46.627273810 (590h54m46.498356845s) [upload=518023]: Done uploading segments...
Dec 30 06:23:46.627299639 (590h54m46.498382684s) [upload=518023]: Upload finished. err: <nil>
Duration: 6.359373953s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [..............................X...X......X............X.....] duration:
32.506442606s trace:
https://jaeger.dev.storj.io/trace/45b769fc71e13448
FAIL (2025-12-30 06:26:22 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
.577697683 (590h53m11.448780728s) [noise=true , address=193.149.164.143:4704 , piece_id=4T2IJIA6WHI5LMIUP5BLJLCD3GRY3A6PXR6G5OWJWUSYZT5ELGHQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578173528 (590h53m11.449256573s) [noise=true , address=64.49.13.27:4965 , piece_id=CO4O5F2IBZAAKBZBV3HWNNOXZOGLNNDPALWMQZ3DEM3MREASVFEA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578542992 (590h53m11.449626027s) [noise=true , address=77.90.31.15:4729 , piece_id=NVMXFEILHGT622IIPYYT4FFHC2WZLHMCZM4VT5WUPHRAPH3DX6TA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578846911 (590h53m11.449929946s) [noise=true , address=145.79.227.43:4177 , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579458044 (590h53m11.450541089s) [noise=true , address=45.91.116.11:4150 , piece_id=E66IDHTJDLYLYJIOWMOIKOOWLXCUNCMBIQ7SQBAD3VTJF6VX5JLQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579891209 (590h53m11.450974245s) [noise=true , address=145.79.227.29:4945 , piece_id=SHL73IDEI54GRCWYNWHRUSJZZ4UUXREAX54DFJLTG5QPZCF5QKIQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579915535 (590h53m11.45099858s) [noise=true , address=145.79.227.30:4772 , piece_id=PVMD5DWRRPVARTSZ6O7HN6MUVTZUYWLYSNQR5GDMNKUJZGXMYPFQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580325466 (590h53m11.451408511s) [noise=true , address=77.90.31.12:4603 , piece_id=2I4L7F2WEGK5OSLODJSJ4672URBA7DUGXWZKTYB6ZNA2TQIGVXWA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580379599 (590h53m11.451462644s) [noise=true , address=77.90.31.14:4671 , piece_id=263TRISJ5ADFTX43ULSSGBAF2KMOXAPISJ56LLE53D74XOEMQ5EA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580692044 (590h53m11.451775089s) [noise=true , address=145.79.227.3:4175 , piece_id=UCV5WQNVWEAMEDBWDNXEVSQEGXPJE7KD6YSWC5A4WXYUKUCK4NLA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580857859 (590h53m11.451940894s) [noise=true , address=77.90.31.19:4860 , piece_id=U2MNFOAD7FHCDBE4FUYOROIP3P33ROJ7K75SF5Z747EFKFTF7RGA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581089210 (590h53m11.452172255s) [noise=true , address=145.79.227.44:4203 , piece_id=WHCKQ2FRLYAZE7Q6HYZNTS4EISVHZNAPDDKH34WN33ALK4MVK2EA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581351320 (590h53m11.452434355s) [noise=true , address=64.49.13.154:4928 , piece_id=SEVBZBMNGURP5TRUEBA4PHRSA6MNANVJUKINALJUWPQWBGKVYNTA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581457091 (590h53m11.452540126s) [noise=true , address=145.79.227.34:4909 , piece_id=3QSWTNR5LGUUAY3N5HJKSPL3TRQ6WO5F5OD4VTXKWHWUXUF7PVVQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581863144 (590h53m11.452946179s) [noise=true , address=145.79.227.33:4881 , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.584010443 (590h53m11.455093478s) [noise=true , address=145.79.227.40:4070 , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.585122791 (590h53m11.456205826s) [noise=true , address=45.91.116.14:4217 , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:12.983305402 (590h53m12.854388447s) [upload=517998]: Next returned nil segment
Dec 30 06:22:12.983321313 (590h53m12.854404348s) [upload=517998]: Waiting for error group managing segments...
Dec 30 06:22:13.379703053 (590h53m13.250786088s) [noise=true , address=64.49.13.27:4965 , piece_id=CO4O5F2IBZAAKBZBV3HWNNOXZOGLNNDPALWMQZ3DEM3MREASVFEA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.418055622 (590h53m13.289138657s) [noise=true , address=145.79.227.3:4175 , piece_id=UCV5WQNVWEAMEDBWDNXEVSQEGXPJE7KD6YSWC5A4WXYUKUCK4NLA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.509034160 (590h53m13.380117205s) [noise=true , address=64.49.13.154:4928 , piece_id=SEVBZBMNGURP5TRUEBA4PHRSA6MNANVJUKINALJUWPQWBGKVYNTA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.642886154 (590h53m13.513969199s) [noise=true , address=77.90.31.14:4671 , piece_id=263TRISJ5ADFTX43ULSSGBAF2KMOXAPISJ56LLE53D74XOEMQ5EA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.788145520 (590h53m13.659228555s) [noise=true , address=77.90.31.12:4603 , piece_id=2I4L7F2WEGK5OSLODJSJ4672URBA7DUGXWZKTYB6ZNA2TQIGVXWA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.858878757 (590h53m13.729961792s) [noise=true , address=145.79.227.49:4369 , piece_id=XUEXGNJJIJ4DUBDNUV7CKF6J5JYHSLZAEPVGEDGBUBY42BIP4XEQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.879550569 (590h53m13.750633614s) [noise=true , address=145.79.227.34:4909 , piece_id=3QSWTNR5LGUUAY3N5HJKSPL3TRQ6WO5F5OD4VTXKWHWUXUF7PVVQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.940897723 (590h53m13.811980769s) [noise=true , address=77.90.31.19:4860 , piece_id=U2MNFOAD7FHCDBE4FUYOROIP3P33ROJ7K75SF5Z747EFKFTF7RGA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.989791936 (590h53m13.860874981s) [noise=true , address=45.91.116.11:4150 , piece_id=E66IDHTJDLYLYJIOWMOIKOOWLXCUNCMBIQ7SQBAD3VTJF6VX5JLQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.289516718 (590h53m14.160599753s) [noise=true , address=145.79.227.44:4203 , piece_id=WHCKQ2FRLYAZE7Q6HYZNTS4EISVHZNAPDDKH34WN33ALK4MVK2EA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.338651537 (590h53m14.209734582s) [noise=true , address=77.90.31.15:4729 , piece_id=NVMXFEILHGT622IIPYYT4FFHC2WZLHMCZM4VT5WUPHRAPH3DX6TA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.344145168 (590h53m14.215228213s) [noise=true , address=145.79.227.30:4772 , piece_id=PVMD5DWRRPVARTSZ6O7HN6MUVTZUYWLYSNQR5GDMNKUJZGXMYPFQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.667232847 (590h53m14.538315892s) [noise=true , address=193.149.164.143:4704 , piece_id=4T2IJIA6WHI5LMIUP5BLJLCD3GRY3A6PXR6G5OWJWUSYZT5ELGHQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.671388990 (590h53m14.542472025s) [noise=true , address=145.79.227.29:4945 , piece_id=SHL73IDEI54GRCWYNWHRUSJZZ4UUXREAX54DFJLTG5QPZCF5QKIQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.671415190 (590h53m14.542498226s) [seg_pos={0 0}, upload=517998]: Segment reached optimal threshold of 14 pieces.
Dec 30 06:22:14.671803600 (590h53m14.542886655s) [noise=true , address=145.79.227.40:4070 , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hU1LwL2gX7rVnrNgzaBCYB37ziqf7s7aUgtF2Uc3antReQ6me): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.671814470 (590h53m14.542897525s) [noise=true , address=145.79.227.40:4070 , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672014932 (590h53m14.543097977s) [noise=true , address=145.79.227.43:4177 , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bhN3cR22QuAYihnnB3x3R97UZRWJ8sFPP8fjQcXZ7qhNZphZG): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672022637 (590h53m14.543105672s) [noise=true , address=145.79.227.43:4177 , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672088933 (590h53m14.543171988s) [noise=true , address=145.79.227.33:4881 , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672127677 (590h53m14.543210742s) [noise=true , address=145.79.227.33:4881 , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672312739 (590h53m14.543395774s) [noise=true , address=45.91.116.14:4217 , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xSXWFiq27Dd2vBDD9vVnhDHVG59UmDKn3hVAhf3oyo5goVNaE): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672319953 (590h53m14.543402988s) [noise=true , address=45.91.116.14:4217 , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672337847 (590h53m14.543420882s) [seg_pos={0 0}, upload=517998]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 06:22:14.672452947 (590h53m14.543535982s) [upload=517998]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:22:14.672458507 (590h53m14.543541542s) [upload=517998]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:22:14.981437069 (590h53m14.852520114s) [upload=517998]: Done uploading segments...
Dec 30 06:22:14.981456546 (590h53m14.852539581s) [upload=517998]: Upload finished. err: <nil>
Duration: 32.506467513s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..............................................X.............] duration:
6.723910979s trace:
https://jaeger.dev.storj.io/trace/deb96436003dee
FAIL (2025-12-30 06:26:20 UTC)
orbiter warning: read tcp 188.166.225.170:52458->45.38.24.2:443: read: connection reset by peer
Duration: 6.723945614s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
17 out of 60 is failed (threshold=3) [....X.XX.X..X..X..X..XX.......X..X......X....X...X..X.X.X...] duration:
6.062241902s trace:
https://jaeger.dev.storj.io/trace/7fa8434019f3dc08
FAIL (2025-12-30 06:26:01 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: FN7D5LQNXZSK3XUYFXEAMDQTZ3DT5ETXGL5BJTQZG2RGFSHV2SJQ) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: N7QZZC6PLJDCPSVL3UE6DQXR4UPUGV2NK3VT533OI2CO2OO77PQQ) context deadline exceeded
======LOGS======
Dec 30 06:23:43.071179046 (590h54m42.942262101s) [upload=518023]: Starting upload
Dec 30 06:23:43.071274326 (590h54m42.942357372s) [upload=518023]: Done starting upload
Dec 30 06:23:43.071313622 (590h54m42.942396667s) [upload=518023]: Uploading segments...
Dec 30 06:23:43.071515386 (590h54m42.942598421s) [upload=518023]: Got next segment. Inline: false
Dec 30 06:23:43.071545433 (590h54m42.942628468s) [upload=518023]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 06:23:43.071551454 (590h54m42.942634499s) [upload=518023]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 06:23:43.320606664 (590h54m43.191689699s) [seg_pos={0 0}, upload=518023]: Begin upload segment...
Dec 30 06:23:43.320697006 (590h54m43.191780041s) [noise=true , address=145.79.227.12:4767 , piece_id=4QQQC5HR4WZHWHIDCZLJHF6LRUBAHREDRC55EEFDKGYSJLAI4OQQ, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.321480387 (590h54m43.192563422s) [seg_pos={0 0}, upload=518023]: Done begin upload segment.
Dec 30 06:23:43.321524281 (590h54m43.192607316s) [noise=true , address=145.79.227.27:4689 , piece_id=DBTRIDJH4JXR422HJK74ARR4HIAQ6SL3ZIPAVDHHLBY4LUFMCV3A, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.321848659 (590h54m43.192931694s) [noise=true , address=77.90.31.15:4697 , piece_id=ACAUVL7V363DYXVSTC4IUVYQQGLLWB76EUXZJCIM2A3IGEL7YYDA, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.321928801 (590h54m43.193011836s) [noise=true , address=145.79.227.38:4978 , piece_id=CSFJ52LVERSSRACZZLAW7JBKDXTSK7UFA5K23WGO5LD5GJ7DCJUA, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.322137569 (590h54m43.193220604s) [noise=true , address=45.91.116.10:4084 , piece_id=FN7D5LQNXZSK3XUYFXEAMDQTZ3DT5ETXGL5BJTQZG2RGFSHV2SJQ, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.322259802 (590h54m43.193342837s) [noise=true , address=145.79.227.15:4875 , piece_id=I7MXKFDGIF7BARQPGRBTVUU6RBPATRBZBTD6UB32KNBCQVR6ACUA, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.322421469 (590h54m43.193504514s) [noise=true , address=77.90.31.13:4627 , piece_id=SHVJ5BTIJXAC4INZYXVOX4SLVISMQ7AHGDMBNZFTWKR4674GH37A, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.322614057 (590h54m43.193697092s) [noise=true , address=145.79.227.41:4087 , piece_id=34KUIE26SDFSXQAKED4ABATH6QFFR3Z2BWGTKNFGEJWAFV3QLJYA, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.322720900 (590h54m43.193803935s) [noise=true , address=145.79.227.16:4911 , piece_id=5OTUFDE34TVGBLEL6DOAODW6MRFNUHX7IYJSFFLNK4TQ5AOIE2HQ, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.323005622 (590h54m43.194088657s) [noise=true , address=77.90.31.14:4662 , piece_id=Z3I7EEBHWDBRRKIDQAOCO5VM7LDY7LM2MMDGVSEM665H5F3TDJHA, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.323728659 (590h54m43.194811694s) [noise=true , address=77.90.31.16:4736 , piece_id=N7QZZC6PLJDCPSVL3UE6DQXR4UPUGV2NK3VT533OI2CO2OO77PQQ, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.324350754 (590h54m43.195433789s) [noise=true , address=145.79.227.9:4366 , piece_id=G2W6PSVI2KWTOVBRNOA4RJUXPVQ6HYXYR3ILZPNBHWYZH5N54FTQ, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.325895645 (590h54m43.19697868s) [noise=true , address=145.79.227.47:4265 , piece_id=IJOEH7UD7Q3LKFUEAMDDCO75VMDMFOE4OXJKQNKSESEV5DGIICBQ, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.326582463 (590h54m43.197665508s) [noise=true , address=45.91.116.9:4261 , piece_id=CQDG2JLVYT22T4NKLQLW3ZGAHDJ4CHJLFSJXZS26FLHRE46MNICQ, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.327825790 (590h54m43.198908825s) [noise=true , address=45.91.116.6:4973 , piece_id=FQDPNZYSZTFTMN5CDGC2TFF3GAAWSYYXNXYTM4KRCN7QSZBSGSQA, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.329280390 (590h54m43.200363435s) [noise=true , address=45.91.116.8:4047 , piece_id=R33VTNE36IROAHKNRCZXHWRRLUJGKEFFWI3GSG2B5IPECDWLG2LQ, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:45.023502686 (590h54m44.894585731s) [upload=518023]: Next returned nil segment
Dec 30 06:23:45.023528155 (590h54m44.8946112s) [upload=518023]: Waiting for error group managing segments...
Dec 30 06:23:45.309378197 (590h54m45.180461242s) [noise=true , address=145.79.227.38:4978 , piece_id=CSFJ52LVERSSRACZZLAW7JBKDXTSK7UFA5K23WGO5LD5GJ7DCJUA, seg_pos={0 0}, upload=518023]: Done uploading piece. err: <nil>
Dec 30 06:23:45.310585435 (590h54m45.18166848s) [noise=true , address=145.79.227.9:4366 , piece_id=G2W6PSVI2KWTOVBRNOA4RJUXPVQ6HYXYR3ILZPNBHWYZH5N54FTQ, seg_pos={0 0}, upload=518023]: Done uploading piece. err: <nil>
Dec 30 06:23:45.313225593 (590h54m45.184308628s) [noise=true , address=145.79.227.16:4911 , piece_id=5OTUFDE34TVGBLEL6DOAODW6MRFNUHX7IYJSFFLNK4TQ5AOIE2HQ, seg_pos={0 0}, upload=518023]: Done uploading piece. err: <nil>
Dec 30 06:23:45.313449610 (590h54m45.184532645s) [noise=true , address=145.79.227.41:4087 , piece_id=34KUIE26SDFSXQAKED4ABATH6QFFR3Z2BWGTKNFGEJWAFV3QLJYA, seg_pos={0 0}, upload=518023]: Done uploading piece. err: <nil>
Dec 30 06:23:45.328340344 (590h54m45.199423379s) [noise=true , address=145.79.227.27:4689 , piece_id=DBTRIDJH4JXR422HJK74ARR4HIAQ6SL3ZIPAVDHHLBY4LUFMCV3A, seg_pos={0 0}, upload=518023]: Done uploading piece. err: <nil>
Dec 30 06:23:45.329301494 (590h54m45.20038479s) [noise=true , address=145.79.227.12:4767 , piece_id=4QQQC5HR4WZHWHIDCZLJHF6LRUBAHREDRC55EEFDKGYSJLAI4OQQ, seg_pos={0 0}, upload=518023]: Done uploading piece. err: <nil>
Dec 30 06:23:45.331819288 (590h54m45.202902323s) [noise=true , address=145.79.227.15:4875 , piece_id=I7MXKFDGIF7BARQPGRBTVUU6RBPATRBZBTD6UB32KNBCQVR6ACUA, seg_pos={0 0}, upload=518023]: Done uploading piece. err: <nil>
Dec 30 06:23:45.348197735 (590h54m45.21928077s) [noise=true , address=145.79.227.47:4265 , piece_id=IJOEH7UD7Q3LKFUEAMDDCO75VMDMFOE4OXJKQNKSESEV5DGIICBQ, seg_pos={0 0}, upload=518023]: Done uploading piece. err: <nil>
Dec 30 06:23:45.583085838 (590h54m45.454168873s) [noise=true , address=77.90.31.15:4697 , piece_id=ACAUVL7V363DYXVSTC4IUVYQQGLLWB76EUXZJCIM2A3IGEL7YYDA, seg_pos={0 0}, upload=518023]: Done uploading piece. err: <nil>
Dec 30 06:23:45.634110076 (590h54m45.505193111s) [noise=true , address=45.91.116.10:4084 , piece_id=FN7D5LQNXZSK3XUYFXEAMDQTZ3DT5ETXGL5BJTQZG2RGFSHV2SJQ, seg_pos={0 0}, upload=518023]: Done uploading piece. err: <nil>
Dec 30 06:23:46.004722348 (590h54m45.875805383s) [noise=true , address=45.91.116.8:4047 , piece_id=R33VTNE36IROAHKNRCZXHWRRLUJGKEFFWI3GSG2B5IPECDWLG2LQ, seg_pos={0 0}, upload=518023]: Done uploading piece. err: <nil>
Dec 30 06:23:46.236932912 (590h54m46.108015957s) [noise=true , address=77.90.31.16:4736 , piece_id=N7QZZC6PLJDCPSVL3UE6DQXR4UPUGV2NK3VT533OI2CO2OO77PQQ, seg_pos={0 0}, upload=518023]: Done uploading piece. err: <nil>
Dec 30 06:23:46.325242185 (590h54m46.19632522s) [noise=true , address=77.90.31.14:4662 , piece_id=Z3I7EEBHWDBRRKIDQAOCO5VM7LDY7LM2MMDGVSEM665H5F3TDJHA, seg_pos={0 0}, upload=518023]: Done uploading piece. err: <nil>
Dec 30 06:23:46.325267093 (590h54m46.196350138s) [seg_pos={0 0}, upload=518023]: Segment reached optimal threshold of 13 pieces.
Dec 30 06:23:46.325517570 (590h54m46.196600615s) [noise=true , address=45.91.116.6:4973 , piece_id=FQDPNZYSZTFTMN5CDGC2TFF3GAAWSYYXNXYTM4KRCN7QSZBSGSQA, seg_pos={0 0}, upload=518023]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF
Dec 30 06:23:46.325527078 (590h54m46.196610114s) [noise=true , address=45.91.116.6:4973 , piece_id=FQDPNZYSZTFTMN5CDGC2TFF3GAAWSYYXNXYTM4KRCN7QSZBSGSQA, seg_pos={0 0}, upload=518023]: Long tail cancellation due to optimal threshold
Dec 30 06:23:46.325646135 (590h54m46.19672919s) [noise=true , address=77.90.31.13:4627 , piece_id=SHVJ5BTIJXAC4INZYXVOX4SLVISMQ7AHGDMBNZFTWKR4674GH37A, seg_pos={0 0}, upload=518023]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:59304->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:59304->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59304->77.90.31.13:4627: use of closed network connection
Dec 30 06:23:46.325676553 (590h54m46.196759588s) [noise=true , address=77.90.31.13:4627 , piece_id=SHVJ5BTIJXAC4INZYXVOX4SLVISMQ7AHGDMBNZFTWKR4674GH37A, seg_pos={0 0}, upload=518023]: Long tail cancellation due to optimal threshold
Dec 30 06:23:46.325887665 (590h54m46.1969707s) [noise=true , address=45.91.116.9:4261 , piece_id=CQDG2JLVYT22T4NKLQLW3ZGAHDJ4CHJLFSJXZS26FLHRE46MNICQ, seg_pos={0 0}, upload=518023]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF
Dec 30 06:23:46.325895149 (590h54m46.196978184s) [noise=true , address=45.91.116.9:4261 , piece_id=CQDG2JLVYT22T4NKLQLW3ZGAHDJ4CHJLFSJXZS26FLHRE46MNICQ, seg_pos={0 0}, upload=518023]: Long tail cancellation due to optimal threshold
Dec 30 06:23:46.325926509 (590h54m46.197009544s) [seg_pos={0 0}, upload=518023]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 06:23:46.326041127 (590h54m46.197124182s) [upload=518023]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:23:46.326050726 (590h54m46.197133761s) [upload=518023]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:23:46.627273810 (590h54m46.498356845s) [upload=518023]: Done uploading segments...
Dec 30 06:23:46.627299639 (590h54m46.498382684s) [upload=518023]: Upload finished. err: <nil>
Duration: 6.062259024s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
18 out of 60 is failed (threshold=3) [X....X.XX.X..X..X..X..XX.......X..X......X....X...X..X.X.X..] duration:
29.522120546s trace:
https://jaeger.dev.storj.io/trace/7fa8434019f3dc08
FAIL (2025-12-30 06:25:55 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: FN7D5LQNXZSK3XUYFXEAMDQTZ3DT5ETXGL5BJTQZG2RGFSHV2SJQ) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: N7QZZC6PLJDCPSVL3UE6DQXR4UPUGV2NK3VT533OI2CO2OO77PQQ) context deadline exceeded
======LOGS======
Dec 30 06:23:43.071179046 (590h54m42.942262101s) [upload=518023]: Starting upload
Dec 30 06:23:43.071274326 (590h54m42.942357372s) [upload=518023]: Done starting upload
Dec 30 06:23:43.071313622 (590h54m42.942396667s) [upload=518023]: Uploading segments...
Dec 30 06:23:43.071515386 (590h54m42.942598421s) [upload=518023]: Got next segment. Inline: false
Dec 30 06:23:43.071545433 (590h54m42.942628468s) [upload=518023]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 06:23:43.071551454 (590h54m42.942634499s) [upload=518023]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 06:23:43.320606664 (590h54m43.191689699s) [seg_pos={0 0}, upload=518023]: Begin upload segment...
Dec 30 06:23:43.320697006 (590h54m43.191780041s) [noise=true , address=145.79.227.12:4767 , piece_id=4QQQC5HR4WZHWHIDCZLJHF6LRUBAHREDRC55EEFDKGYSJLAI4OQQ, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.321480387 (590h54m43.192563422s) [seg_pos={0 0}, upload=518023]: Done begin upload segment.
Dec 30 06:23:43.321524281 (590h54m43.192607316s) [noise=true , address=145.79.227.27:4689 , piece_id=DBTRIDJH4JXR422HJK74ARR4HIAQ6SL3ZIPAVDHHLBY4LUFMCV3A, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.321848659 (590h54m43.192931694s) [noise=true , address=77.90.31.15:4697 , piece_id=ACAUVL7V363DYXVSTC4IUVYQQGLLWB76EUXZJCIM2A3IGEL7YYDA, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.321928801 (590h54m43.193011836s) [noise=true , address=145.79.227.38:4978 , piece_id=CSFJ52LVERSSRACZZLAW7JBKDXTSK7UFA5K23WGO5LD5GJ7DCJUA, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.322137569 (590h54m43.193220604s) [noise=true , address=45.91.116.10:4084 , piece_id=FN7D5LQNXZSK3XUYFXEAMDQTZ3DT5ETXGL5BJTQZG2RGFSHV2SJQ, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.322259802 (590h54m43.193342837s) [noise=true , address=145.79.227.15:4875 , piece_id=I7MXKFDGIF7BARQPGRBTVUU6RBPATRBZBTD6UB32KNBCQVR6ACUA, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.322421469 (590h54m43.193504514s) [noise=true , address=77.90.31.13:4627 , piece_id=SHVJ5BTIJXAC4INZYXVOX4SLVISMQ7AHGDMBNZFTWKR4674GH37A, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.322614057 (590h54m43.193697092s) [noise=true , address=145.79.227.41:4087 , piece_id=34KUIE26SDFSXQAKED4ABATH6QFFR3Z2BWGTKNFGEJWAFV3QLJYA, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.322720900 (590h54m43.193803935s) [noise=true , address=145.79.227.16:4911 , piece_id=5OTUFDE34TVGBLEL6DOAODW6MRFNUHX7IYJSFFLNK4TQ5AOIE2HQ, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.323005622 (590h54m43.194088657s) [noise=true , address=77.90.31.14:4662 , piece_id=Z3I7EEBHWDBRRKIDQAOCO5VM7LDY7LM2MMDGVSEM665H5F3TDJHA, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.323728659 (590h54m43.194811694s) [noise=true , address=77.90.31.16:4736 , piece_id=N7QZZC6PLJDCPSVL3UE6DQXR4UPUGV2NK3VT533OI2CO2OO77PQQ, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.324350754 (590h54m43.195433789s) [noise=true , address=145.79.227.9:4366 , piece_id=G2W6PSVI2KWTOVBRNOA4RJUXPVQ6HYXYR3ILZPNBHWYZH5N54FTQ, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.325895645 (590h54m43.19697868s) [noise=true , address=145.79.227.47:4265 , piece_id=IJOEH7UD7Q3LKFUEAMDDCO75VMDMFOE4OXJKQNKSESEV5DGIICBQ, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.326582463 (590h54m43.197665508s) [noise=true , address=45.91.116.9:4261 , piece_id=CQDG2JLVYT22T4NKLQLW3ZGAHDJ4CHJLFSJXZS26FLHRE46MNICQ, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.327825790 (590h54m43.198908825s) [noise=true , address=45.91.116.6:4973 , piece_id=FQDPNZYSZTFTMN5CDGC2TFF3GAAWSYYXNXYTM4KRCN7QSZBSGSQA, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.329280390 (590h54m43.200363435s) [noise=true , address=45.91.116.8:4047 , piece_id=R33VTNE36IROAHKNRCZXHWRRLUJGKEFFWI3GSG2B5IPECDWLG2LQ, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:45.023502686 (590h54m44.894585731s) [upload=518023]: Next returned nil segment
Dec 30 06:23:45.023528155 (590h54m44.8946112s) [upload=518023]: Waiting for error group managing segments...
Dec 30 06:23:45.309378197 (590h54m45.180461242s) [noise=true , address=145.79.227.38:4978 , piece_id=CSFJ52LVERSSRACZZLAW7JBKDXTSK7UFA5K23WGO5LD5GJ7DCJUA, seg_pos={0 0}, upload=518023]: Done uploading piece. err: <nil>
Dec 30 06:23:45.310585435 (590h54m45.18166848s) [noise=true , address=145.79.227.9:4366 , piece_id=G2W6PSVI2KWTOVBRNOA4RJUXPVQ6HYXYR3ILZPNBHWYZH5N54FTQ, seg_pos={0 0}, upload=518023]: Done uploading piece. err: <nil>
Dec 30 06:23:45.313225593 (590h54m45.184308628s) [noise=true , address=145.79.227.16:4911 , piece_id=5OTUFDE34TVGBLEL6DOAODW6MRFNUHX7IYJSFFLNK4TQ5AOIE2HQ, seg_pos={0 0}, upload=518023]: Done uploading piece. err: <nil>
Dec 30 06:23:45.313449610 (590h54m45.184532645s) [noise=true , address=145.79.227.41:4087 , piece_id=34KUIE26SDFSXQAKED4ABATH6QFFR3Z2BWGTKNFGEJWAFV3QLJYA, seg_pos={0 0}, upload=518023]: Done uploading piece. err: <nil>
Dec 30 06:23:45.328340344 (590h54m45.199423379s) [noise=true , address=145.79.227.27:4689 , piece_id=DBTRIDJH4JXR422HJK74ARR4HIAQ6SL3ZIPAVDHHLBY4LUFMCV3A, seg_pos={0 0}, upload=518023]: Done uploading piece. err: <nil>
Dec 30 06:23:45.329301494 (590h54m45.20038479s) [noise=true , address=145.79.227.12:4767 , piece_id=4QQQC5HR4WZHWHIDCZLJHF6LRUBAHREDRC55EEFDKGYSJLAI4OQQ, seg_pos={0 0}, upload=518023]: Done uploading piece. err: <nil>
Dec 30 06:23:45.331819288 (590h54m45.202902323s) [noise=true , address=145.79.227.15:4875 , piece_id=I7MXKFDGIF7BARQPGRBTVUU6RBPATRBZBTD6UB32KNBCQVR6ACUA, seg_pos={0 0}, upload=518023]: Done uploading piece. err: <nil>
Dec 30 06:23:45.348197735 (590h54m45.21928077s) [noise=true , address=145.79.227.47:4265 , piece_id=IJOEH7UD7Q3LKFUEAMDDCO75VMDMFOE4OXJKQNKSESEV5DGIICBQ, seg_pos={0 0}, upload=518023]: Done uploading piece. err: <nil>
Dec 30 06:23:45.583085838 (590h54m45.454168873s) [noise=true , address=77.90.31.15:4697 , piece_id=ACAUVL7V363DYXVSTC4IUVYQQGLLWB76EUXZJCIM2A3IGEL7YYDA, seg_pos={0 0}, upload=518023]: Done uploading piece. err: <nil>
Dec 30 06:23:45.634110076 (590h54m45.505193111s) [noise=true , address=45.91.116.10:4084 , piece_id=FN7D5LQNXZSK3XUYFXEAMDQTZ3DT5ETXGL5BJTQZG2RGFSHV2SJQ, seg_pos={0 0}, upload=518023]: Done uploading piece. err: <nil>
Dec 30 06:23:46.004722348 (590h54m45.875805383s) [noise=true , address=45.91.116.8:4047 , piece_id=R33VTNE36IROAHKNRCZXHWRRLUJGKEFFWI3GSG2B5IPECDWLG2LQ, seg_pos={0 0}, upload=518023]: Done uploading piece. err: <nil>
Dec 30 06:23:46.236932912 (590h54m46.108015957s) [noise=true , address=77.90.31.16:4736 , piece_id=N7QZZC6PLJDCPSVL3UE6DQXR4UPUGV2NK3VT533OI2CO2OO77PQQ, seg_pos={0 0}, upload=518023]: Done uploading piece. err: <nil>
Dec 30 06:23:46.325242185 (590h54m46.19632522s) [noise=true , address=77.90.31.14:4662 , piece_id=Z3I7EEBHWDBRRKIDQAOCO5VM7LDY7LM2MMDGVSEM665H5F3TDJHA, seg_pos={0 0}, upload=518023]: Done uploading piece. err: <nil>
Dec 30 06:23:46.325267093 (590h54m46.196350138s) [seg_pos={0 0}, upload=518023]: Segment reached optimal threshold of 13 pieces.
Dec 30 06:23:46.325517570 (590h54m46.196600615s) [noise=true , address=45.91.116.6:4973 , piece_id=FQDPNZYSZTFTMN5CDGC2TFF3GAAWSYYXNXYTM4KRCN7QSZBSGSQA, seg_pos={0 0}, upload=518023]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF
Dec 30 06:23:46.325527078 (590h54m46.196610114s) [noise=true , address=45.91.116.6:4973 , piece_id=FQDPNZYSZTFTMN5CDGC2TFF3GAAWSYYXNXYTM4KRCN7QSZBSGSQA, seg_pos={0 0}, upload=518023]: Long tail cancellation due to optimal threshold
Dec 30 06:23:46.325646135 (590h54m46.19672919s) [noise=true , address=77.90.31.13:4627 , piece_id=SHVJ5BTIJXAC4INZYXVOX4SLVISMQ7AHGDMBNZFTWKR4674GH37A, seg_pos={0 0}, upload=518023]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:59304->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:59304->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59304->77.90.31.13:4627: use of closed network connection
Dec 30 06:23:46.325676553 (590h54m46.196759588s) [noise=true , address=77.90.31.13:4627 , piece_id=SHVJ5BTIJXAC4INZYXVOX4SLVISMQ7AHGDMBNZFTWKR4674GH37A, seg_pos={0 0}, upload=518023]: Long tail cancellation due to optimal threshold
Dec 30 06:23:46.325887665 (590h54m46.1969707s) [noise=true , address=45.91.116.9:4261 , piece_id=CQDG2JLVYT22T4NKLQLW3ZGAHDJ4CHJLFSJXZS26FLHRE46MNICQ, seg_pos={0 0}, upload=518023]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF
Dec 30 06:23:46.325895149 (590h54m46.196978184s) [noise=true , address=45.91.116.9:4261 , piece_id=CQDG2JLVYT22T4NKLQLW3ZGAHDJ4CHJLFSJXZS26FLHRE46MNICQ, seg_pos={0 0}, upload=518023]: Long tail cancellation due to optimal threshold
Dec 30 06:23:46.325926509 (590h54m46.197009544s) [seg_pos={0 0}, upload=518023]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 06:23:46.326041127 (590h54m46.197124182s) [upload=518023]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:23:46.326050726 (590h54m46.197133761s) [upload=518023]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:23:46.627273810 (590h54m46.498356845s) [upload=518023]: Done uploading segments...
Dec 30 06:23:46.627299639 (590h54m46.498382684s) [upload=518023]: Upload finished. err: <nil>
Duration: 29.522151076s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [...............................X...X......X............X....] duration:
35.067167857s trace:
https://jaeger.dev.storj.io/trace/45b769fc71e13448
FAIL (2025-12-30 06:25:50 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
.577697683 (590h53m11.448780728s) [noise=true , address=193.149.164.143:4704 , piece_id=4T2IJIA6WHI5LMIUP5BLJLCD3GRY3A6PXR6G5OWJWUSYZT5ELGHQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578173528 (590h53m11.449256573s) [noise=true , address=64.49.13.27:4965 , piece_id=CO4O5F2IBZAAKBZBV3HWNNOXZOGLNNDPALWMQZ3DEM3MREASVFEA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578542992 (590h53m11.449626027s) [noise=true , address=77.90.31.15:4729 , piece_id=NVMXFEILHGT622IIPYYT4FFHC2WZLHMCZM4VT5WUPHRAPH3DX6TA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578846911 (590h53m11.449929946s) [noise=true , address=145.79.227.43:4177 , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579458044 (590h53m11.450541089s) [noise=true , address=45.91.116.11:4150 , piece_id=E66IDHTJDLYLYJIOWMOIKOOWLXCUNCMBIQ7SQBAD3VTJF6VX5JLQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579891209 (590h53m11.450974245s) [noise=true , address=145.79.227.29:4945 , piece_id=SHL73IDEI54GRCWYNWHRUSJZZ4UUXREAX54DFJLTG5QPZCF5QKIQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579915535 (590h53m11.45099858s) [noise=true , address=145.79.227.30:4772 , piece_id=PVMD5DWRRPVARTSZ6O7HN6MUVTZUYWLYSNQR5GDMNKUJZGXMYPFQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580325466 (590h53m11.451408511s) [noise=true , address=77.90.31.12:4603 , piece_id=2I4L7F2WEGK5OSLODJSJ4672URBA7DUGXWZKTYB6ZNA2TQIGVXWA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580379599 (590h53m11.451462644s) [noise=true , address=77.90.31.14:4671 , piece_id=263TRISJ5ADFTX43ULSSGBAF2KMOXAPISJ56LLE53D74XOEMQ5EA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580692044 (590h53m11.451775089s) [noise=true , address=145.79.227.3:4175 , piece_id=UCV5WQNVWEAMEDBWDNXEVSQEGXPJE7KD6YSWC5A4WXYUKUCK4NLA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580857859 (590h53m11.451940894s) [noise=true , address=77.90.31.19:4860 , piece_id=U2MNFOAD7FHCDBE4FUYOROIP3P33ROJ7K75SF5Z747EFKFTF7RGA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581089210 (590h53m11.452172255s) [noise=true , address=145.79.227.44:4203 , piece_id=WHCKQ2FRLYAZE7Q6HYZNTS4EISVHZNAPDDKH34WN33ALK4MVK2EA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581351320 (590h53m11.452434355s) [noise=true , address=64.49.13.154:4928 , piece_id=SEVBZBMNGURP5TRUEBA4PHRSA6MNANVJUKINALJUWPQWBGKVYNTA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581457091 (590h53m11.452540126s) [noise=true , address=145.79.227.34:4909 , piece_id=3QSWTNR5LGUUAY3N5HJKSPL3TRQ6WO5F5OD4VTXKWHWUXUF7PVVQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581863144 (590h53m11.452946179s) [noise=true , address=145.79.227.33:4881 , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.584010443 (590h53m11.455093478s) [noise=true , address=145.79.227.40:4070 , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.585122791 (590h53m11.456205826s) [noise=true , address=45.91.116.14:4217 , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:12.983305402 (590h53m12.854388447s) [upload=517998]: Next returned nil segment
Dec 30 06:22:12.983321313 (590h53m12.854404348s) [upload=517998]: Waiting for error group managing segments...
Dec 30 06:22:13.379703053 (590h53m13.250786088s) [noise=true , address=64.49.13.27:4965 , piece_id=CO4O5F2IBZAAKBZBV3HWNNOXZOGLNNDPALWMQZ3DEM3MREASVFEA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.418055622 (590h53m13.289138657s) [noise=true , address=145.79.227.3:4175 , piece_id=UCV5WQNVWEAMEDBWDNXEVSQEGXPJE7KD6YSWC5A4WXYUKUCK4NLA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.509034160 (590h53m13.380117205s) [noise=true , address=64.49.13.154:4928 , piece_id=SEVBZBMNGURP5TRUEBA4PHRSA6MNANVJUKINALJUWPQWBGKVYNTA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.642886154 (590h53m13.513969199s) [noise=true , address=77.90.31.14:4671 , piece_id=263TRISJ5ADFTX43ULSSGBAF2KMOXAPISJ56LLE53D74XOEMQ5EA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.788145520 (590h53m13.659228555s) [noise=true , address=77.90.31.12:4603 , piece_id=2I4L7F2WEGK5OSLODJSJ4672URBA7DUGXWZKTYB6ZNA2TQIGVXWA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.858878757 (590h53m13.729961792s) [noise=true , address=145.79.227.49:4369 , piece_id=XUEXGNJJIJ4DUBDNUV7CKF6J5JYHSLZAEPVGEDGBUBY42BIP4XEQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.879550569 (590h53m13.750633614s) [noise=true , address=145.79.227.34:4909 , piece_id=3QSWTNR5LGUUAY3N5HJKSPL3TRQ6WO5F5OD4VTXKWHWUXUF7PVVQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.940897723 (590h53m13.811980769s) [noise=true , address=77.90.31.19:4860 , piece_id=U2MNFOAD7FHCDBE4FUYOROIP3P33ROJ7K75SF5Z747EFKFTF7RGA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.989791936 (590h53m13.860874981s) [noise=true , address=45.91.116.11:4150 , piece_id=E66IDHTJDLYLYJIOWMOIKOOWLXCUNCMBIQ7SQBAD3VTJF6VX5JLQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.289516718 (590h53m14.160599753s) [noise=true , address=145.79.227.44:4203 , piece_id=WHCKQ2FRLYAZE7Q6HYZNTS4EISVHZNAPDDKH34WN33ALK4MVK2EA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.338651537 (590h53m14.209734582s) [noise=true , address=77.90.31.15:4729 , piece_id=NVMXFEILHGT622IIPYYT4FFHC2WZLHMCZM4VT5WUPHRAPH3DX6TA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.344145168 (590h53m14.215228213s) [noise=true , address=145.79.227.30:4772 , piece_id=PVMD5DWRRPVARTSZ6O7HN6MUVTZUYWLYSNQR5GDMNKUJZGXMYPFQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.667232847 (590h53m14.538315892s) [noise=true , address=193.149.164.143:4704 , piece_id=4T2IJIA6WHI5LMIUP5BLJLCD3GRY3A6PXR6G5OWJWUSYZT5ELGHQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.671388990 (590h53m14.542472025s) [noise=true , address=145.79.227.29:4945 , piece_id=SHL73IDEI54GRCWYNWHRUSJZZ4UUXREAX54DFJLTG5QPZCF5QKIQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.671415190 (590h53m14.542498226s) [seg_pos={0 0}, upload=517998]: Segment reached optimal threshold of 14 pieces.
Dec 30 06:22:14.671803600 (590h53m14.542886655s) [noise=true , address=145.79.227.40:4070 , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hU1LwL2gX7rVnrNgzaBCYB37ziqf7s7aUgtF2Uc3antReQ6me): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.671814470 (590h53m14.542897525s) [noise=true , address=145.79.227.40:4070 , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672014932 (590h53m14.543097977s) [noise=true , address=145.79.227.43:4177 , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bhN3cR22QuAYihnnB3x3R97UZRWJ8sFPP8fjQcXZ7qhNZphZG): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672022637 (590h53m14.543105672s) [noise=true , address=145.79.227.43:4177 , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672088933 (590h53m14.543171988s) [noise=true , address=145.79.227.33:4881 , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672127677 (590h53m14.543210742s) [noise=true , address=145.79.227.33:4881 , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672312739 (590h53m14.543395774s) [noise=true , address=45.91.116.14:4217 , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xSXWFiq27Dd2vBDD9vVnhDHVG59UmDKn3hVAhf3oyo5goVNaE): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672319953 (590h53m14.543402988s) [noise=true , address=45.91.116.14:4217 , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672337847 (590h53m14.543420882s) [seg_pos={0 0}, upload=517998]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 06:22:14.672452947 (590h53m14.543535982s) [upload=517998]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:22:14.672458507 (590h53m14.543541542s) [upload=517998]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:22:14.981437069 (590h53m14.852520114s) [upload=517998]: Done uploading segments...
Dec 30 06:22:14.981456546 (590h53m14.852539581s) [upload=517998]: Upload finished. err: <nil>
Duration: 35.06722736s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [..............................X...........X..............XX.] duration:
3.391080471s trace:
https://jaeger.dev.storj.io/trace/2e49128c973fa898
FAIL (2025-12-30 06:25:34 UTC)
orbiter warning: context deadline exceeded
Duration: 3.391110078s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...............................................X............] duration:
7.00528134s trace:
https://jaeger.dev.storj.io/trace/deb96436003dee
FAIL (2025-12-30 06:25:30 UTC)
orbiter warning: read tcp 188.166.225.170:52458->45.38.24.2:443: read: connection reset by peer
Duration: 7.005310556s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [................................................X...........] duration:
6.328025167s trace:
https://jaeger.dev.storj.io/trace/deb96436003dee
FAIL (2025-12-30 06:24:51 UTC)
orbiter warning: read tcp 188.166.225.170:52458->45.38.24.2:443: read: connection reset by peer
Duration: 6.328047519s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
18 out of 60 is failed (threshold=3) [.X....X.XX.X..X..X..X..XX.......X..X......X....X...X..X.X.X.] duration:
6.338029961s trace:
https://jaeger.dev.storj.io/trace/7fa8434019f3dc08
FAIL (2025-12-30 06:24:49 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: FN7D5LQNXZSK3XUYFXEAMDQTZ3DT5ETXGL5BJTQZG2RGFSHV2SJQ) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: N7QZZC6PLJDCPSVL3UE6DQXR4UPUGV2NK3VT533OI2CO2OO77PQQ) context deadline exceeded
======LOGS======
Dec 30 06:23:43.071179046 (590h54m42.942262101s) [upload=518023]: Starting upload
Dec 30 06:23:43.071274326 (590h54m42.942357372s) [upload=518023]: Done starting upload
Dec 30 06:23:43.071313622 (590h54m42.942396667s) [upload=518023]: Uploading segments...
Dec 30 06:23:43.071515386 (590h54m42.942598421s) [upload=518023]: Got next segment. Inline: false
Dec 30 06:23:43.071545433 (590h54m42.942628468s) [upload=518023]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 06:23:43.071551454 (590h54m42.942634499s) [upload=518023]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 06:23:43.320606664 (590h54m43.191689699s) [seg_pos={0 0}, upload=518023]: Begin upload segment...
Dec 30 06:23:43.320697006 (590h54m43.191780041s) [noise=true , address=145.79.227.12:4767 , piece_id=4QQQC5HR4WZHWHIDCZLJHF6LRUBAHREDRC55EEFDKGYSJLAI4OQQ, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.321480387 (590h54m43.192563422s) [seg_pos={0 0}, upload=518023]: Done begin upload segment.
Dec 30 06:23:43.321524281 (590h54m43.192607316s) [noise=true , address=145.79.227.27:4689 , piece_id=DBTRIDJH4JXR422HJK74ARR4HIAQ6SL3ZIPAVDHHLBY4LUFMCV3A, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.321848659 (590h54m43.192931694s) [noise=true , address=77.90.31.15:4697 , piece_id=ACAUVL7V363DYXVSTC4IUVYQQGLLWB76EUXZJCIM2A3IGEL7YYDA, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.321928801 (590h54m43.193011836s) [noise=true , address=145.79.227.38:4978 , piece_id=CSFJ52LVERSSRACZZLAW7JBKDXTSK7UFA5K23WGO5LD5GJ7DCJUA, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.322137569 (590h54m43.193220604s) [noise=true , address=45.91.116.10:4084 , piece_id=FN7D5LQNXZSK3XUYFXEAMDQTZ3DT5ETXGL5BJTQZG2RGFSHV2SJQ, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.322259802 (590h54m43.193342837s) [noise=true , address=145.79.227.15:4875 , piece_id=I7MXKFDGIF7BARQPGRBTVUU6RBPATRBZBTD6UB32KNBCQVR6ACUA, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.322421469 (590h54m43.193504514s) [noise=true , address=77.90.31.13:4627 , piece_id=SHVJ5BTIJXAC4INZYXVOX4SLVISMQ7AHGDMBNZFTWKR4674GH37A, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.322614057 (590h54m43.193697092s) [noise=true , address=145.79.227.41:4087 , piece_id=34KUIE26SDFSXQAKED4ABATH6QFFR3Z2BWGTKNFGEJWAFV3QLJYA, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.322720900 (590h54m43.193803935s) [noise=true , address=145.79.227.16:4911 , piece_id=5OTUFDE34TVGBLEL6DOAODW6MRFNUHX7IYJSFFLNK4TQ5AOIE2HQ, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.323005622 (590h54m43.194088657s) [noise=true , address=77.90.31.14:4662 , piece_id=Z3I7EEBHWDBRRKIDQAOCO5VM7LDY7LM2MMDGVSEM665H5F3TDJHA, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.323728659 (590h54m43.194811694s) [noise=true , address=77.90.31.16:4736 , piece_id=N7QZZC6PLJDCPSVL3UE6DQXR4UPUGV2NK3VT533OI2CO2OO77PQQ, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.324350754 (590h54m43.195433789s) [noise=true , address=145.79.227.9:4366 , piece_id=G2W6PSVI2KWTOVBRNOA4RJUXPVQ6HYXYR3ILZPNBHWYZH5N54FTQ, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.325895645 (590h54m43.19697868s) [noise=true , address=145.79.227.47:4265 , piece_id=IJOEH7UD7Q3LKFUEAMDDCO75VMDMFOE4OXJKQNKSESEV5DGIICBQ, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.326582463 (590h54m43.197665508s) [noise=true , address=45.91.116.9:4261 , piece_id=CQDG2JLVYT22T4NKLQLW3ZGAHDJ4CHJLFSJXZS26FLHRE46MNICQ, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.327825790 (590h54m43.198908825s) [noise=true , address=45.91.116.6:4973 , piece_id=FQDPNZYSZTFTMN5CDGC2TFF3GAAWSYYXNXYTM4KRCN7QSZBSGSQA, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.329280390 (590h54m43.200363435s) [noise=true , address=45.91.116.8:4047 , piece_id=R33VTNE36IROAHKNRCZXHWRRLUJGKEFFWI3GSG2B5IPECDWLG2LQ, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:45.023502686 (590h54m44.894585731s) [upload=518023]: Next returned nil segment
Dec 30 06:23:45.023528155 (590h54m44.8946112s) [upload=518023]: Waiting for error group managing segments...
Dec 30 06:23:45.309378197 (590h54m45.180461242s) [noise=true , address=145.79.227.38:4978 , piece_id=CSFJ52LVERSSRACZZLAW7JBKDXTSK7UFA5K23WGO5LD5GJ7DCJUA, seg_pos={0 0}, upload=518023]: Done uploading piece. err: <nil>
Dec 30 06:23:45.310585435 (590h54m45.18166848s) [noise=true , address=145.79.227.9:4366 , piece_id=G2W6PSVI2KWTOVBRNOA4RJUXPVQ6HYXYR3ILZPNBHWYZH5N54FTQ, seg_pos={0 0}, upload=518023]: Done uploading piece. err: <nil>
Dec 30 06:23:45.313225593 (590h54m45.184308628s) [noise=true , address=145.79.227.16:4911 , piece_id=5OTUFDE34TVGBLEL6DOAODW6MRFNUHX7IYJSFFLNK4TQ5AOIE2HQ, seg_pos={0 0}, upload=518023]: Done uploading piece. err: <nil>
Dec 30 06:23:45.313449610 (590h54m45.184532645s) [noise=true , address=145.79.227.41:4087 , piece_id=34KUIE26SDFSXQAKED4ABATH6QFFR3Z2BWGTKNFGEJWAFV3QLJYA, seg_pos={0 0}, upload=518023]: Done uploading piece. err: <nil>
Dec 30 06:23:45.328340344 (590h54m45.199423379s) [noise=true , address=145.79.227.27:4689 , piece_id=DBTRIDJH4JXR422HJK74ARR4HIAQ6SL3ZIPAVDHHLBY4LUFMCV3A, seg_pos={0 0}, upload=518023]: Done uploading piece. err: <nil>
Dec 30 06:23:45.329301494 (590h54m45.20038479s) [noise=true , address=145.79.227.12:4767 , piece_id=4QQQC5HR4WZHWHIDCZLJHF6LRUBAHREDRC55EEFDKGYSJLAI4OQQ, seg_pos={0 0}, upload=518023]: Done uploading piece. err: <nil>
Dec 30 06:23:45.331819288 (590h54m45.202902323s) [noise=true , address=145.79.227.15:4875 , piece_id=I7MXKFDGIF7BARQPGRBTVUU6RBPATRBZBTD6UB32KNBCQVR6ACUA, seg_pos={0 0}, upload=518023]: Done uploading piece. err: <nil>
Dec 30 06:23:45.348197735 (590h54m45.21928077s) [noise=true , address=145.79.227.47:4265 , piece_id=IJOEH7UD7Q3LKFUEAMDDCO75VMDMFOE4OXJKQNKSESEV5DGIICBQ, seg_pos={0 0}, upload=518023]: Done uploading piece. err: <nil>
Dec 30 06:23:45.583085838 (590h54m45.454168873s) [noise=true , address=77.90.31.15:4697 , piece_id=ACAUVL7V363DYXVSTC4IUVYQQGLLWB76EUXZJCIM2A3IGEL7YYDA, seg_pos={0 0}, upload=518023]: Done uploading piece. err: <nil>
Dec 30 06:23:45.634110076 (590h54m45.505193111s) [noise=true , address=45.91.116.10:4084 , piece_id=FN7D5LQNXZSK3XUYFXEAMDQTZ3DT5ETXGL5BJTQZG2RGFSHV2SJQ, seg_pos={0 0}, upload=518023]: Done uploading piece. err: <nil>
Dec 30 06:23:46.004722348 (590h54m45.875805383s) [noise=true , address=45.91.116.8:4047 , piece_id=R33VTNE36IROAHKNRCZXHWRRLUJGKEFFWI3GSG2B5IPECDWLG2LQ, seg_pos={0 0}, upload=518023]: Done uploading piece. err: <nil>
Dec 30 06:23:46.236932912 (590h54m46.108015957s) [noise=true , address=77.90.31.16:4736 , piece_id=N7QZZC6PLJDCPSVL3UE6DQXR4UPUGV2NK3VT533OI2CO2OO77PQQ, seg_pos={0 0}, upload=518023]: Done uploading piece. err: <nil>
Dec 30 06:23:46.325242185 (590h54m46.19632522s) [noise=true , address=77.90.31.14:4662 , piece_id=Z3I7EEBHWDBRRKIDQAOCO5VM7LDY7LM2MMDGVSEM665H5F3TDJHA, seg_pos={0 0}, upload=518023]: Done uploading piece. err: <nil>
Dec 30 06:23:46.325267093 (590h54m46.196350138s) [seg_pos={0 0}, upload=518023]: Segment reached optimal threshold of 13 pieces.
Dec 30 06:23:46.325517570 (590h54m46.196600615s) [noise=true , address=45.91.116.6:4973 , piece_id=FQDPNZYSZTFTMN5CDGC2TFF3GAAWSYYXNXYTM4KRCN7QSZBSGSQA, seg_pos={0 0}, upload=518023]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF
Dec 30 06:23:46.325527078 (590h54m46.196610114s) [noise=true , address=45.91.116.6:4973 , piece_id=FQDPNZYSZTFTMN5CDGC2TFF3GAAWSYYXNXYTM4KRCN7QSZBSGSQA, seg_pos={0 0}, upload=518023]: Long tail cancellation due to optimal threshold
Dec 30 06:23:46.325646135 (590h54m46.19672919s) [noise=true , address=77.90.31.13:4627 , piece_id=SHVJ5BTIJXAC4INZYXVOX4SLVISMQ7AHGDMBNZFTWKR4674GH37A, seg_pos={0 0}, upload=518023]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:59304->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:59304->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59304->77.90.31.13:4627: use of closed network connection
Dec 30 06:23:46.325676553 (590h54m46.196759588s) [noise=true , address=77.90.31.13:4627 , piece_id=SHVJ5BTIJXAC4INZYXVOX4SLVISMQ7AHGDMBNZFTWKR4674GH37A, seg_pos={0 0}, upload=518023]: Long tail cancellation due to optimal threshold
Dec 30 06:23:46.325887665 (590h54m46.1969707s) [noise=true , address=45.91.116.9:4261 , piece_id=CQDG2JLVYT22T4NKLQLW3ZGAHDJ4CHJLFSJXZS26FLHRE46MNICQ, seg_pos={0 0}, upload=518023]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF
Dec 30 06:23:46.325895149 (590h54m46.196978184s) [noise=true , address=45.91.116.9:4261 , piece_id=CQDG2JLVYT22T4NKLQLW3ZGAHDJ4CHJLFSJXZS26FLHRE46MNICQ, seg_pos={0 0}, upload=518023]: Long tail cancellation due to optimal threshold
Dec 30 06:23:46.325926509 (590h54m46.197009544s) [seg_pos={0 0}, upload=518023]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 06:23:46.326041127 (590h54m46.197124182s) [upload=518023]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:23:46.326050726 (590h54m46.197133761s) [upload=518023]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:23:46.627273810 (590h54m46.498356845s) [upload=518023]: Done uploading segments...
Dec 30 06:23:46.627299639 (590h54m46.498382684s) [upload=518023]: Upload finished. err: <nil>
Duration: 6.338052995s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [................................X...X......X............X...] duration:
28.181273535s trace:
https://jaeger.dev.storj.io/trace/45b769fc71e13448
FAIL (2025-12-30 06:24:49 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
.577697683 (590h53m11.448780728s) [noise=true , address=193.149.164.143:4704 , piece_id=4T2IJIA6WHI5LMIUP5BLJLCD3GRY3A6PXR6G5OWJWUSYZT5ELGHQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578173528 (590h53m11.449256573s) [noise=true , address=64.49.13.27:4965 , piece_id=CO4O5F2IBZAAKBZBV3HWNNOXZOGLNNDPALWMQZ3DEM3MREASVFEA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578542992 (590h53m11.449626027s) [noise=true , address=77.90.31.15:4729 , piece_id=NVMXFEILHGT622IIPYYT4FFHC2WZLHMCZM4VT5WUPHRAPH3DX6TA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578846911 (590h53m11.449929946s) [noise=true , address=145.79.227.43:4177 , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579458044 (590h53m11.450541089s) [noise=true , address=45.91.116.11:4150 , piece_id=E66IDHTJDLYLYJIOWMOIKOOWLXCUNCMBIQ7SQBAD3VTJF6VX5JLQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579891209 (590h53m11.450974245s) [noise=true , address=145.79.227.29:4945 , piece_id=SHL73IDEI54GRCWYNWHRUSJZZ4UUXREAX54DFJLTG5QPZCF5QKIQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579915535 (590h53m11.45099858s) [noise=true , address=145.79.227.30:4772 , piece_id=PVMD5DWRRPVARTSZ6O7HN6MUVTZUYWLYSNQR5GDMNKUJZGXMYPFQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580325466 (590h53m11.451408511s) [noise=true , address=77.90.31.12:4603 , piece_id=2I4L7F2WEGK5OSLODJSJ4672URBA7DUGXWZKTYB6ZNA2TQIGVXWA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580379599 (590h53m11.451462644s) [noise=true , address=77.90.31.14:4671 , piece_id=263TRISJ5ADFTX43ULSSGBAF2KMOXAPISJ56LLE53D74XOEMQ5EA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580692044 (590h53m11.451775089s) [noise=true , address=145.79.227.3:4175 , piece_id=UCV5WQNVWEAMEDBWDNXEVSQEGXPJE7KD6YSWC5A4WXYUKUCK4NLA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580857859 (590h53m11.451940894s) [noise=true , address=77.90.31.19:4860 , piece_id=U2MNFOAD7FHCDBE4FUYOROIP3P33ROJ7K75SF5Z747EFKFTF7RGA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581089210 (590h53m11.452172255s) [noise=true , address=145.79.227.44:4203 , piece_id=WHCKQ2FRLYAZE7Q6HYZNTS4EISVHZNAPDDKH34WN33ALK4MVK2EA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581351320 (590h53m11.452434355s) [noise=true , address=64.49.13.154:4928 , piece_id=SEVBZBMNGURP5TRUEBA4PHRSA6MNANVJUKINALJUWPQWBGKVYNTA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581457091 (590h53m11.452540126s) [noise=true , address=145.79.227.34:4909 , piece_id=3QSWTNR5LGUUAY3N5HJKSPL3TRQ6WO5F5OD4VTXKWHWUXUF7PVVQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581863144 (590h53m11.452946179s) [noise=true , address=145.79.227.33:4881 , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.584010443 (590h53m11.455093478s) [noise=true , address=145.79.227.40:4070 , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.585122791 (590h53m11.456205826s) [noise=true , address=45.91.116.14:4217 , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:12.983305402 (590h53m12.854388447s) [upload=517998]: Next returned nil segment
Dec 30 06:22:12.983321313 (590h53m12.854404348s) [upload=517998]: Waiting for error group managing segments...
Dec 30 06:22:13.379703053 (590h53m13.250786088s) [noise=true , address=64.49.13.27:4965 , piece_id=CO4O5F2IBZAAKBZBV3HWNNOXZOGLNNDPALWMQZ3DEM3MREASVFEA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.418055622 (590h53m13.289138657s) [noise=true , address=145.79.227.3:4175 , piece_id=UCV5WQNVWEAMEDBWDNXEVSQEGXPJE7KD6YSWC5A4WXYUKUCK4NLA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.509034160 (590h53m13.380117205s) [noise=true , address=64.49.13.154:4928 , piece_id=SEVBZBMNGURP5TRUEBA4PHRSA6MNANVJUKINALJUWPQWBGKVYNTA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.642886154 (590h53m13.513969199s) [noise=true , address=77.90.31.14:4671 , piece_id=263TRISJ5ADFTX43ULSSGBAF2KMOXAPISJ56LLE53D74XOEMQ5EA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.788145520 (590h53m13.659228555s) [noise=true , address=77.90.31.12:4603 , piece_id=2I4L7F2WEGK5OSLODJSJ4672URBA7DUGXWZKTYB6ZNA2TQIGVXWA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.858878757 (590h53m13.729961792s) [noise=true , address=145.79.227.49:4369 , piece_id=XUEXGNJJIJ4DUBDNUV7CKF6J5JYHSLZAEPVGEDGBUBY42BIP4XEQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.879550569 (590h53m13.750633614s) [noise=true , address=145.79.227.34:4909 , piece_id=3QSWTNR5LGUUAY3N5HJKSPL3TRQ6WO5F5OD4VTXKWHWUXUF7PVVQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.940897723 (590h53m13.811980769s) [noise=true , address=77.90.31.19:4860 , piece_id=U2MNFOAD7FHCDBE4FUYOROIP3P33ROJ7K75SF5Z747EFKFTF7RGA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.989791936 (590h53m13.860874981s) [noise=true , address=45.91.116.11:4150 , piece_id=E66IDHTJDLYLYJIOWMOIKOOWLXCUNCMBIQ7SQBAD3VTJF6VX5JLQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.289516718 (590h53m14.160599753s) [noise=true , address=145.79.227.44:4203 , piece_id=WHCKQ2FRLYAZE7Q6HYZNTS4EISVHZNAPDDKH34WN33ALK4MVK2EA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.338651537 (590h53m14.209734582s) [noise=true , address=77.90.31.15:4729 , piece_id=NVMXFEILHGT622IIPYYT4FFHC2WZLHMCZM4VT5WUPHRAPH3DX6TA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.344145168 (590h53m14.215228213s) [noise=true , address=145.79.227.30:4772 , piece_id=PVMD5DWRRPVARTSZ6O7HN6MUVTZUYWLYSNQR5GDMNKUJZGXMYPFQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.667232847 (590h53m14.538315892s) [noise=true , address=193.149.164.143:4704 , piece_id=4T2IJIA6WHI5LMIUP5BLJLCD3GRY3A6PXR6G5OWJWUSYZT5ELGHQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.671388990 (590h53m14.542472025s) [noise=true , address=145.79.227.29:4945 , piece_id=SHL73IDEI54GRCWYNWHRUSJZZ4UUXREAX54DFJLTG5QPZCF5QKIQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.671415190 (590h53m14.542498226s) [seg_pos={0 0}, upload=517998]: Segment reached optimal threshold of 14 pieces.
Dec 30 06:22:14.671803600 (590h53m14.542886655s) [noise=true , address=145.79.227.40:4070 , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hU1LwL2gX7rVnrNgzaBCYB37ziqf7s7aUgtF2Uc3antReQ6me): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.671814470 (590h53m14.542897525s) [noise=true , address=145.79.227.40:4070 , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672014932 (590h53m14.543097977s) [noise=true , address=145.79.227.43:4177 , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bhN3cR22QuAYihnnB3x3R97UZRWJ8sFPP8fjQcXZ7qhNZphZG): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672022637 (590h53m14.543105672s) [noise=true , address=145.79.227.43:4177 , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672088933 (590h53m14.543171988s) [noise=true , address=145.79.227.33:4881 , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672127677 (590h53m14.543210742s) [noise=true , address=145.79.227.33:4881 , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672312739 (590h53m14.543395774s) [noise=true , address=45.91.116.14:4217 , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xSXWFiq27Dd2vBDD9vVnhDHVG59UmDKn3hVAhf3oyo5goVNaE): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672319953 (590h53m14.543402988s) [noise=true , address=45.91.116.14:4217 , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672337847 (590h53m14.543420882s) [seg_pos={0 0}, upload=517998]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 06:22:14.672452947 (590h53m14.543535982s) [upload=517998]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:22:14.672458507 (590h53m14.543541542s) [upload=517998]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:22:14.981437069 (590h53m14.852520114s) [upload=517998]: Done uploading segments...
Dec 30 06:22:14.981456546 (590h53m14.852539581s) [upload=517998]: Upload finished. err: <nil>
Duration: 28.181473806s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
4 out of 60 is failed (threshold=6) [...............................X...........X..............XX] duration:
5.001294077s trace:
https://jaeger.dev.storj.io/trace/2e49128c973fa898
FAIL (2025-12-30 06:24:31 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001318553s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [.................................X...X......X............X..] duration:
15.205689695s trace:
https://jaeger.dev.storj.io/trace/45b769fc71e13448
FAIL (2025-12-30 06:24:15 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
.577697683 (590h53m11.448780728s) [noise=true , address=193.149.164.143:4704 , piece_id=4T2IJIA6WHI5LMIUP5BLJLCD3GRY3A6PXR6G5OWJWUSYZT5ELGHQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578173528 (590h53m11.449256573s) [noise=true , address=64.49.13.27:4965 , piece_id=CO4O5F2IBZAAKBZBV3HWNNOXZOGLNNDPALWMQZ3DEM3MREASVFEA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578542992 (590h53m11.449626027s) [noise=true , address=77.90.31.15:4729 , piece_id=NVMXFEILHGT622IIPYYT4FFHC2WZLHMCZM4VT5WUPHRAPH3DX6TA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578846911 (590h53m11.449929946s) [noise=true , address=145.79.227.43:4177 , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579458044 (590h53m11.450541089s) [noise=true , address=45.91.116.11:4150 , piece_id=E66IDHTJDLYLYJIOWMOIKOOWLXCUNCMBIQ7SQBAD3VTJF6VX5JLQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579891209 (590h53m11.450974245s) [noise=true , address=145.79.227.29:4945 , piece_id=SHL73IDEI54GRCWYNWHRUSJZZ4UUXREAX54DFJLTG5QPZCF5QKIQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579915535 (590h53m11.45099858s) [noise=true , address=145.79.227.30:4772 , piece_id=PVMD5DWRRPVARTSZ6O7HN6MUVTZUYWLYSNQR5GDMNKUJZGXMYPFQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580325466 (590h53m11.451408511s) [noise=true , address=77.90.31.12:4603 , piece_id=2I4L7F2WEGK5OSLODJSJ4672URBA7DUGXWZKTYB6ZNA2TQIGVXWA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580379599 (590h53m11.451462644s) [noise=true , address=77.90.31.14:4671 , piece_id=263TRISJ5ADFTX43ULSSGBAF2KMOXAPISJ56LLE53D74XOEMQ5EA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580692044 (590h53m11.451775089s) [noise=true , address=145.79.227.3:4175 , piece_id=UCV5WQNVWEAMEDBWDNXEVSQEGXPJE7KD6YSWC5A4WXYUKUCK4NLA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580857859 (590h53m11.451940894s) [noise=true , address=77.90.31.19:4860 , piece_id=U2MNFOAD7FHCDBE4FUYOROIP3P33ROJ7K75SF5Z747EFKFTF7RGA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581089210 (590h53m11.452172255s) [noise=true , address=145.79.227.44:4203 , piece_id=WHCKQ2FRLYAZE7Q6HYZNTS4EISVHZNAPDDKH34WN33ALK4MVK2EA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581351320 (590h53m11.452434355s) [noise=true , address=64.49.13.154:4928 , piece_id=SEVBZBMNGURP5TRUEBA4PHRSA6MNANVJUKINALJUWPQWBGKVYNTA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581457091 (590h53m11.452540126s) [noise=true , address=145.79.227.34:4909 , piece_id=3QSWTNR5LGUUAY3N5HJKSPL3TRQ6WO5F5OD4VTXKWHWUXUF7PVVQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581863144 (590h53m11.452946179s) [noise=true , address=145.79.227.33:4881 , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.584010443 (590h53m11.455093478s) [noise=true , address=145.79.227.40:4070 , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.585122791 (590h53m11.456205826s) [noise=true , address=45.91.116.14:4217 , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:12.983305402 (590h53m12.854388447s) [upload=517998]: Next returned nil segment
Dec 30 06:22:12.983321313 (590h53m12.854404348s) [upload=517998]: Waiting for error group managing segments...
Dec 30 06:22:13.379703053 (590h53m13.250786088s) [noise=true , address=64.49.13.27:4965 , piece_id=CO4O5F2IBZAAKBZBV3HWNNOXZOGLNNDPALWMQZ3DEM3MREASVFEA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.418055622 (590h53m13.289138657s) [noise=true , address=145.79.227.3:4175 , piece_id=UCV5WQNVWEAMEDBWDNXEVSQEGXPJE7KD6YSWC5A4WXYUKUCK4NLA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.509034160 (590h53m13.380117205s) [noise=true , address=64.49.13.154:4928 , piece_id=SEVBZBMNGURP5TRUEBA4PHRSA6MNANVJUKINALJUWPQWBGKVYNTA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.642886154 (590h53m13.513969199s) [noise=true , address=77.90.31.14:4671 , piece_id=263TRISJ5ADFTX43ULSSGBAF2KMOXAPISJ56LLE53D74XOEMQ5EA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.788145520 (590h53m13.659228555s) [noise=true , address=77.90.31.12:4603 , piece_id=2I4L7F2WEGK5OSLODJSJ4672URBA7DUGXWZKTYB6ZNA2TQIGVXWA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.858878757 (590h53m13.729961792s) [noise=true , address=145.79.227.49:4369 , piece_id=XUEXGNJJIJ4DUBDNUV7CKF6J5JYHSLZAEPVGEDGBUBY42BIP4XEQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.879550569 (590h53m13.750633614s) [noise=true , address=145.79.227.34:4909 , piece_id=3QSWTNR5LGUUAY3N5HJKSPL3TRQ6WO5F5OD4VTXKWHWUXUF7PVVQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.940897723 (590h53m13.811980769s) [noise=true , address=77.90.31.19:4860 , piece_id=U2MNFOAD7FHCDBE4FUYOROIP3P33ROJ7K75SF5Z747EFKFTF7RGA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.989791936 (590h53m13.860874981s) [noise=true , address=45.91.116.11:4150 , piece_id=E66IDHTJDLYLYJIOWMOIKOOWLXCUNCMBIQ7SQBAD3VTJF6VX5JLQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.289516718 (590h53m14.160599753s) [noise=true , address=145.79.227.44:4203 , piece_id=WHCKQ2FRLYAZE7Q6HYZNTS4EISVHZNAPDDKH34WN33ALK4MVK2EA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.338651537 (590h53m14.209734582s) [noise=true , address=77.90.31.15:4729 , piece_id=NVMXFEILHGT622IIPYYT4FFHC2WZLHMCZM4VT5WUPHRAPH3DX6TA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.344145168 (590h53m14.215228213s) [noise=true , address=145.79.227.30:4772 , piece_id=PVMD5DWRRPVARTSZ6O7HN6MUVTZUYWLYSNQR5GDMNKUJZGXMYPFQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.667232847 (590h53m14.538315892s) [noise=true , address=193.149.164.143:4704 , piece_id=4T2IJIA6WHI5LMIUP5BLJLCD3GRY3A6PXR6G5OWJWUSYZT5ELGHQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.671388990 (590h53m14.542472025s) [noise=true , address=145.79.227.29:4945 , piece_id=SHL73IDEI54GRCWYNWHRUSJZZ4UUXREAX54DFJLTG5QPZCF5QKIQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.671415190 (590h53m14.542498226s) [seg_pos={0 0}, upload=517998]: Segment reached optimal threshold of 14 pieces.
Dec 30 06:22:14.671803600 (590h53m14.542886655s) [noise=true , address=145.79.227.40:4070 , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hU1LwL2gX7rVnrNgzaBCYB37ziqf7s7aUgtF2Uc3antReQ6me): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.671814470 (590h53m14.542897525s) [noise=true , address=145.79.227.40:4070 , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672014932 (590h53m14.543097977s) [noise=true , address=145.79.227.43:4177 , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bhN3cR22QuAYihnnB3x3R97UZRWJ8sFPP8fjQcXZ7qhNZphZG): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672022637 (590h53m14.543105672s) [noise=true , address=145.79.227.43:4177 , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672088933 (590h53m14.543171988s) [noise=true , address=145.79.227.33:4881 , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672127677 (590h53m14.543210742s) [noise=true , address=145.79.227.33:4881 , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672312739 (590h53m14.543395774s) [noise=true , address=45.91.116.14:4217 , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xSXWFiq27Dd2vBDD9vVnhDHVG59UmDKn3hVAhf3oyo5goVNaE): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672319953 (590h53m14.543402988s) [noise=true , address=45.91.116.14:4217 , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672337847 (590h53m14.543420882s) [seg_pos={0 0}, upload=517998]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 06:22:14.672452947 (590h53m14.543535982s) [upload=517998]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:22:14.672458507 (590h53m14.543541542s) [upload=517998]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:22:14.981437069 (590h53m14.852520114s) [upload=517998]: Done uploading segments...
Dec 30 06:22:14.981456546 (590h53m14.852539581s) [upload=517998]: Upload finished. err: <nil>
Duration: 15.205716196s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
18 out of 60 is failed (threshold=3) [..X....X.XX.X..X..X..X..XX.......X..X......X....X...X..X.X.X] duration:
30.003272753s trace:
https://jaeger.dev.storj.io/trace/7fa8434019f3dc08
FAIL (2025-12-30 06:24:12 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: FN7D5LQNXZSK3XUYFXEAMDQTZ3DT5ETXGL5BJTQZG2RGFSHV2SJQ) context deadline exceeded; piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: N7QZZC6PLJDCPSVL3UE6DQXR4UPUGV2NK3VT533OI2CO2OO77PQQ) context deadline exceeded
======LOGS======
Dec 30 06:23:43.071179046 (590h54m42.942262101s) [upload=518023]: Starting upload
Dec 30 06:23:43.071274326 (590h54m42.942357372s) [upload=518023]: Done starting upload
Dec 30 06:23:43.071313622 (590h54m42.942396667s) [upload=518023]: Uploading segments...
Dec 30 06:23:43.071515386 (590h54m42.942598421s) [upload=518023]: Got next segment. Inline: false
Dec 30 06:23:43.071545433 (590h54m42.942628468s) [upload=518023]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 06:23:43.071551454 (590h54m42.942634499s) [upload=518023]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 06:23:43.320606664 (590h54m43.191689699s) [seg_pos={0 0}, upload=518023]: Begin upload segment...
Dec 30 06:23:43.320697006 (590h54m43.191780041s) [noise=true , address=145.79.227.12:4767 , piece_id=4QQQC5HR4WZHWHIDCZLJHF6LRUBAHREDRC55EEFDKGYSJLAI4OQQ, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.321480387 (590h54m43.192563422s) [seg_pos={0 0}, upload=518023]: Done begin upload segment.
Dec 30 06:23:43.321524281 (590h54m43.192607316s) [noise=true , address=145.79.227.27:4689 , piece_id=DBTRIDJH4JXR422HJK74ARR4HIAQ6SL3ZIPAVDHHLBY4LUFMCV3A, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.321848659 (590h54m43.192931694s) [noise=true , address=77.90.31.15:4697 , piece_id=ACAUVL7V363DYXVSTC4IUVYQQGLLWB76EUXZJCIM2A3IGEL7YYDA, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.321928801 (590h54m43.193011836s) [noise=true , address=145.79.227.38:4978 , piece_id=CSFJ52LVERSSRACZZLAW7JBKDXTSK7UFA5K23WGO5LD5GJ7DCJUA, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.322137569 (590h54m43.193220604s) [noise=true , address=45.91.116.10:4084 , piece_id=FN7D5LQNXZSK3XUYFXEAMDQTZ3DT5ETXGL5BJTQZG2RGFSHV2SJQ, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.322259802 (590h54m43.193342837s) [noise=true , address=145.79.227.15:4875 , piece_id=I7MXKFDGIF7BARQPGRBTVUU6RBPATRBZBTD6UB32KNBCQVR6ACUA, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.322421469 (590h54m43.193504514s) [noise=true , address=77.90.31.13:4627 , piece_id=SHVJ5BTIJXAC4INZYXVOX4SLVISMQ7AHGDMBNZFTWKR4674GH37A, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.322614057 (590h54m43.193697092s) [noise=true , address=145.79.227.41:4087 , piece_id=34KUIE26SDFSXQAKED4ABATH6QFFR3Z2BWGTKNFGEJWAFV3QLJYA, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.322720900 (590h54m43.193803935s) [noise=true , address=145.79.227.16:4911 , piece_id=5OTUFDE34TVGBLEL6DOAODW6MRFNUHX7IYJSFFLNK4TQ5AOIE2HQ, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.323005622 (590h54m43.194088657s) [noise=true , address=77.90.31.14:4662 , piece_id=Z3I7EEBHWDBRRKIDQAOCO5VM7LDY7LM2MMDGVSEM665H5F3TDJHA, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.323728659 (590h54m43.194811694s) [noise=true , address=77.90.31.16:4736 , piece_id=N7QZZC6PLJDCPSVL3UE6DQXR4UPUGV2NK3VT533OI2CO2OO77PQQ, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.324350754 (590h54m43.195433789s) [noise=true , address=145.79.227.9:4366 , piece_id=G2W6PSVI2KWTOVBRNOA4RJUXPVQ6HYXYR3ILZPNBHWYZH5N54FTQ, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.325895645 (590h54m43.19697868s) [noise=true , address=145.79.227.47:4265 , piece_id=IJOEH7UD7Q3LKFUEAMDDCO75VMDMFOE4OXJKQNKSESEV5DGIICBQ, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.326582463 (590h54m43.197665508s) [noise=true , address=45.91.116.9:4261 , piece_id=CQDG2JLVYT22T4NKLQLW3ZGAHDJ4CHJLFSJXZS26FLHRE46MNICQ, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.327825790 (590h54m43.198908825s) [noise=true , address=45.91.116.6:4973 , piece_id=FQDPNZYSZTFTMN5CDGC2TFF3GAAWSYYXNXYTM4KRCN7QSZBSGSQA, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:43.329280390 (590h54m43.200363435s) [noise=true , address=45.91.116.8:4047 , piece_id=R33VTNE36IROAHKNRCZXHWRRLUJGKEFFWI3GSG2B5IPECDWLG2LQ, seg_pos={0 0}, upload=518023]: Uploading piece...
Dec 30 06:23:45.023502686 (590h54m44.894585731s) [upload=518023]: Next returned nil segment
Dec 30 06:23:45.023528155 (590h54m44.8946112s) [upload=518023]: Waiting for error group managing segments...
Dec 30 06:23:45.309378197 (590h54m45.180461242s) [noise=true , address=145.79.227.38:4978 , piece_id=CSFJ52LVERSSRACZZLAW7JBKDXTSK7UFA5K23WGO5LD5GJ7DCJUA, seg_pos={0 0}, upload=518023]: Done uploading piece. err: <nil>
Dec 30 06:23:45.310585435 (590h54m45.18166848s) [noise=true , address=145.79.227.9:4366 , piece_id=G2W6PSVI2KWTOVBRNOA4RJUXPVQ6HYXYR3ILZPNBHWYZH5N54FTQ, seg_pos={0 0}, upload=518023]: Done uploading piece. err: <nil>
Dec 30 06:23:45.313225593 (590h54m45.184308628s) [noise=true , address=145.79.227.16:4911 , piece_id=5OTUFDE34TVGBLEL6DOAODW6MRFNUHX7IYJSFFLNK4TQ5AOIE2HQ, seg_pos={0 0}, upload=518023]: Done uploading piece. err: <nil>
Dec 30 06:23:45.313449610 (590h54m45.184532645s) [noise=true , address=145.79.227.41:4087 , piece_id=34KUIE26SDFSXQAKED4ABATH6QFFR3Z2BWGTKNFGEJWAFV3QLJYA, seg_pos={0 0}, upload=518023]: Done uploading piece. err: <nil>
Dec 30 06:23:45.328340344 (590h54m45.199423379s) [noise=true , address=145.79.227.27:4689 , piece_id=DBTRIDJH4JXR422HJK74ARR4HIAQ6SL3ZIPAVDHHLBY4LUFMCV3A, seg_pos={0 0}, upload=518023]: Done uploading piece. err: <nil>
Dec 30 06:23:45.329301494 (590h54m45.20038479s) [noise=true , address=145.79.227.12:4767 , piece_id=4QQQC5HR4WZHWHIDCZLJHF6LRUBAHREDRC55EEFDKGYSJLAI4OQQ, seg_pos={0 0}, upload=518023]: Done uploading piece. err: <nil>
Dec 30 06:23:45.331819288 (590h54m45.202902323s) [noise=true , address=145.79.227.15:4875 , piece_id=I7MXKFDGIF7BARQPGRBTVUU6RBPATRBZBTD6UB32KNBCQVR6ACUA, seg_pos={0 0}, upload=518023]: Done uploading piece. err: <nil>
Dec 30 06:23:45.348197735 (590h54m45.21928077s) [noise=true , address=145.79.227.47:4265 , piece_id=IJOEH7UD7Q3LKFUEAMDDCO75VMDMFOE4OXJKQNKSESEV5DGIICBQ, seg_pos={0 0}, upload=518023]: Done uploading piece. err: <nil>
Dec 30 06:23:45.583085838 (590h54m45.454168873s) [noise=true , address=77.90.31.15:4697 , piece_id=ACAUVL7V363DYXVSTC4IUVYQQGLLWB76EUXZJCIM2A3IGEL7YYDA, seg_pos={0 0}, upload=518023]: Done uploading piece. err: <nil>
Dec 30 06:23:45.634110076 (590h54m45.505193111s) [noise=true , address=45.91.116.10:4084 , piece_id=FN7D5LQNXZSK3XUYFXEAMDQTZ3DT5ETXGL5BJTQZG2RGFSHV2SJQ, seg_pos={0 0}, upload=518023]: Done uploading piece. err: <nil>
Dec 30 06:23:46.004722348 (590h54m45.875805383s) [noise=true , address=45.91.116.8:4047 , piece_id=R33VTNE36IROAHKNRCZXHWRRLUJGKEFFWI3GSG2B5IPECDWLG2LQ, seg_pos={0 0}, upload=518023]: Done uploading piece. err: <nil>
Dec 30 06:23:46.236932912 (590h54m46.108015957s) [noise=true , address=77.90.31.16:4736 , piece_id=N7QZZC6PLJDCPSVL3UE6DQXR4UPUGV2NK3VT533OI2CO2OO77PQQ, seg_pos={0 0}, upload=518023]: Done uploading piece. err: <nil>
Dec 30 06:23:46.325242185 (590h54m46.19632522s) [noise=true , address=77.90.31.14:4662 , piece_id=Z3I7EEBHWDBRRKIDQAOCO5VM7LDY7LM2MMDGVSEM665H5F3TDJHA, seg_pos={0 0}, upload=518023]: Done uploading piece. err: <nil>
Dec 30 06:23:46.325267093 (590h54m46.196350138s) [seg_pos={0 0}, upload=518023]: Segment reached optimal threshold of 13 pieces.
Dec 30 06:23:46.325517570 (590h54m46.196600615s) [noise=true , address=45.91.116.6:4973 , piece_id=FQDPNZYSZTFTMN5CDGC2TFF3GAAWSYYXNXYTM4KRCN7QSZBSGSQA, seg_pos={0 0}, upload=518023]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: expected piece hash; context canceled; EOF
Dec 30 06:23:46.325527078 (590h54m46.196610114s) [noise=true , address=45.91.116.6:4973 , piece_id=FQDPNZYSZTFTMN5CDGC2TFF3GAAWSYYXNXYTM4KRCN7QSZBSGSQA, seg_pos={0 0}, upload=518023]: Long tail cancellation due to optimal threshold
Dec 30 06:23:46.325646135 (590h54m46.19672919s) [noise=true , address=77.90.31.13:4627 , piece_id=SHVJ5BTIJXAC4INZYXVOX4SLVISMQ7AHGDMBNZFTWKR4674GH37A, seg_pos={0 0}, upload=518023]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB): protocol: write tcp 188.166.225.170:59304->77.90.31.13:4627: use of closed network connection; write tcp 188.166.225.170:59304->77.90.31.13:4627: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59304->77.90.31.13:4627: use of closed network connection
Dec 30 06:23:46.325676553 (590h54m46.196759588s) [noise=true , address=77.90.31.13:4627 , piece_id=SHVJ5BTIJXAC4INZYXVOX4SLVISMQ7AHGDMBNZFTWKR4674GH37A, seg_pos={0 0}, upload=518023]: Long tail cancellation due to optimal threshold
Dec 30 06:23:46.325887665 (590h54m46.1969707s) [noise=true , address=45.91.116.9:4261 , piece_id=CQDG2JLVYT22T4NKLQLW3ZGAHDJ4CHJLFSJXZS26FLHRE46MNICQ, seg_pos={0 0}, upload=518023]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF
Dec 30 06:23:46.325895149 (590h54m46.196978184s) [noise=true , address=45.91.116.9:4261 , piece_id=CQDG2JLVYT22T4NKLQLW3ZGAHDJ4CHJLFSJXZS26FLHRE46MNICQ, seg_pos={0 0}, upload=518023]: Long tail cancellation due to optimal threshold
Dec 30 06:23:46.325926509 (590h54m46.197009544s) [seg_pos={0 0}, upload=518023]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 06:23:46.326041127 (590h54m46.197124182s) [upload=518023]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:23:46.326050726 (590h54m46.197133761s) [upload=518023]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:23:46.627273810 (590h54m46.498356845s) [upload=518023]: Done uploading segments...
Dec 30 06:23:46.627299639 (590h54m46.498382684s) [upload=518023]: Upload finished. err: <nil>
Duration: 30.003298992s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=6) [................................X...........X..............X] duration:
5.000619267s trace:
https://jaeger.dev.storj.io/trace/514fe706deac12e8
FAIL (2025-12-30 06:24:00 UTC)
orbiter warning: context deadline exceeded
Duration: 5.000642742s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.................................................X..........] duration:
7.265777214s trace:
https://jaeger.dev.storj.io/trace/deb96436003dee
FAIL (2025-12-30 06:23:58 UTC)
orbiter warning: read tcp 188.166.225.170:52458->45.38.24.2:443: read: connection reset by peer
Duration: 7.265804655s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [..................................X...X......X............X.] duration:
31.131744219s trace:
https://jaeger.dev.storj.io/trace/45b769fc71e13448
FAIL (2025-12-30 06:23:42 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
.577697683 (590h53m11.448780728s) [noise=true , address=193.149.164.143:4704 , piece_id=4T2IJIA6WHI5LMIUP5BLJLCD3GRY3A6PXR6G5OWJWUSYZT5ELGHQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578173528 (590h53m11.449256573s) [noise=true , address=64.49.13.27:4965 , piece_id=CO4O5F2IBZAAKBZBV3HWNNOXZOGLNNDPALWMQZ3DEM3MREASVFEA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578542992 (590h53m11.449626027s) [noise=true , address=77.90.31.15:4729 , piece_id=NVMXFEILHGT622IIPYYT4FFHC2WZLHMCZM4VT5WUPHRAPH3DX6TA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578846911 (590h53m11.449929946s) [noise=true , address=145.79.227.43:4177 , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579458044 (590h53m11.450541089s) [noise=true , address=45.91.116.11:4150 , piece_id=E66IDHTJDLYLYJIOWMOIKOOWLXCUNCMBIQ7SQBAD3VTJF6VX5JLQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579891209 (590h53m11.450974245s) [noise=true , address=145.79.227.29:4945 , piece_id=SHL73IDEI54GRCWYNWHRUSJZZ4UUXREAX54DFJLTG5QPZCF5QKIQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579915535 (590h53m11.45099858s) [noise=true , address=145.79.227.30:4772 , piece_id=PVMD5DWRRPVARTSZ6O7HN6MUVTZUYWLYSNQR5GDMNKUJZGXMYPFQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580325466 (590h53m11.451408511s) [noise=true , address=77.90.31.12:4603 , piece_id=2I4L7F2WEGK5OSLODJSJ4672URBA7DUGXWZKTYB6ZNA2TQIGVXWA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580379599 (590h53m11.451462644s) [noise=true , address=77.90.31.14:4671 , piece_id=263TRISJ5ADFTX43ULSSGBAF2KMOXAPISJ56LLE53D74XOEMQ5EA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580692044 (590h53m11.451775089s) [noise=true , address=145.79.227.3:4175 , piece_id=UCV5WQNVWEAMEDBWDNXEVSQEGXPJE7KD6YSWC5A4WXYUKUCK4NLA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580857859 (590h53m11.451940894s) [noise=true , address=77.90.31.19:4860 , piece_id=U2MNFOAD7FHCDBE4FUYOROIP3P33ROJ7K75SF5Z747EFKFTF7RGA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581089210 (590h53m11.452172255s) [noise=true , address=145.79.227.44:4203 , piece_id=WHCKQ2FRLYAZE7Q6HYZNTS4EISVHZNAPDDKH34WN33ALK4MVK2EA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581351320 (590h53m11.452434355s) [noise=true , address=64.49.13.154:4928 , piece_id=SEVBZBMNGURP5TRUEBA4PHRSA6MNANVJUKINALJUWPQWBGKVYNTA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581457091 (590h53m11.452540126s) [noise=true , address=145.79.227.34:4909 , piece_id=3QSWTNR5LGUUAY3N5HJKSPL3TRQ6WO5F5OD4VTXKWHWUXUF7PVVQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581863144 (590h53m11.452946179s) [noise=true , address=145.79.227.33:4881 , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.584010443 (590h53m11.455093478s) [noise=true , address=145.79.227.40:4070 , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.585122791 (590h53m11.456205826s) [noise=true , address=45.91.116.14:4217 , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:12.983305402 (590h53m12.854388447s) [upload=517998]: Next returned nil segment
Dec 30 06:22:12.983321313 (590h53m12.854404348s) [upload=517998]: Waiting for error group managing segments...
Dec 30 06:22:13.379703053 (590h53m13.250786088s) [noise=true , address=64.49.13.27:4965 , piece_id=CO4O5F2IBZAAKBZBV3HWNNOXZOGLNNDPALWMQZ3DEM3MREASVFEA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.418055622 (590h53m13.289138657s) [noise=true , address=145.79.227.3:4175 , piece_id=UCV5WQNVWEAMEDBWDNXEVSQEGXPJE7KD6YSWC5A4WXYUKUCK4NLA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.509034160 (590h53m13.380117205s) [noise=true , address=64.49.13.154:4928 , piece_id=SEVBZBMNGURP5TRUEBA4PHRSA6MNANVJUKINALJUWPQWBGKVYNTA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.642886154 (590h53m13.513969199s) [noise=true , address=77.90.31.14:4671 , piece_id=263TRISJ5ADFTX43ULSSGBAF2KMOXAPISJ56LLE53D74XOEMQ5EA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.788145520 (590h53m13.659228555s) [noise=true , address=77.90.31.12:4603 , piece_id=2I4L7F2WEGK5OSLODJSJ4672URBA7DUGXWZKTYB6ZNA2TQIGVXWA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.858878757 (590h53m13.729961792s) [noise=true , address=145.79.227.49:4369 , piece_id=XUEXGNJJIJ4DUBDNUV7CKF6J5JYHSLZAEPVGEDGBUBY42BIP4XEQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.879550569 (590h53m13.750633614s) [noise=true , address=145.79.227.34:4909 , piece_id=3QSWTNR5LGUUAY3N5HJKSPL3TRQ6WO5F5OD4VTXKWHWUXUF7PVVQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.940897723 (590h53m13.811980769s) [noise=true , address=77.90.31.19:4860 , piece_id=U2MNFOAD7FHCDBE4FUYOROIP3P33ROJ7K75SF5Z747EFKFTF7RGA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.989791936 (590h53m13.860874981s) [noise=true , address=45.91.116.11:4150 , piece_id=E66IDHTJDLYLYJIOWMOIKOOWLXCUNCMBIQ7SQBAD3VTJF6VX5JLQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.289516718 (590h53m14.160599753s) [noise=true , address=145.79.227.44:4203 , piece_id=WHCKQ2FRLYAZE7Q6HYZNTS4EISVHZNAPDDKH34WN33ALK4MVK2EA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.338651537 (590h53m14.209734582s) [noise=true , address=77.90.31.15:4729 , piece_id=NVMXFEILHGT622IIPYYT4FFHC2WZLHMCZM4VT5WUPHRAPH3DX6TA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.344145168 (590h53m14.215228213s) [noise=true , address=145.79.227.30:4772 , piece_id=PVMD5DWRRPVARTSZ6O7HN6MUVTZUYWLYSNQR5GDMNKUJZGXMYPFQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.667232847 (590h53m14.538315892s) [noise=true , address=193.149.164.143:4704 , piece_id=4T2IJIA6WHI5LMIUP5BLJLCD3GRY3A6PXR6G5OWJWUSYZT5ELGHQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.671388990 (590h53m14.542472025s) [noise=true , address=145.79.227.29:4945 , piece_id=SHL73IDEI54GRCWYNWHRUSJZZ4UUXREAX54DFJLTG5QPZCF5QKIQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.671415190 (590h53m14.542498226s) [seg_pos={0 0}, upload=517998]: Segment reached optimal threshold of 14 pieces.
Dec 30 06:22:14.671803600 (590h53m14.542886655s) [noise=true , address=145.79.227.40:4070 , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hU1LwL2gX7rVnrNgzaBCYB37ziqf7s7aUgtF2Uc3antReQ6me): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.671814470 (590h53m14.542897525s) [noise=true , address=145.79.227.40:4070 , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672014932 (590h53m14.543097977s) [noise=true , address=145.79.227.43:4177 , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bhN3cR22QuAYihnnB3x3R97UZRWJ8sFPP8fjQcXZ7qhNZphZG): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672022637 (590h53m14.543105672s) [noise=true , address=145.79.227.43:4177 , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672088933 (590h53m14.543171988s) [noise=true , address=145.79.227.33:4881 , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672127677 (590h53m14.543210742s) [noise=true , address=145.79.227.33:4881 , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672312739 (590h53m14.543395774s) [noise=true , address=45.91.116.14:4217 , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xSXWFiq27Dd2vBDD9vVnhDHVG59UmDKn3hVAhf3oyo5goVNaE): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672319953 (590h53m14.543402988s) [noise=true , address=45.91.116.14:4217 , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672337847 (590h53m14.543420882s) [seg_pos={0 0}, upload=517998]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 06:22:14.672452947 (590h53m14.543535982s) [upload=517998]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:22:14.672458507 (590h53m14.543541542s) [upload=517998]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:22:14.981437069 (590h53m14.852520114s) [upload=517998]: Done uploading segments...
Dec 30 06:22:14.981456546 (590h53m14.852539581s) [upload=517998]: Upload finished. err: <nil>
Duration: 31.131759569s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.................................X...........X..............] duration:
3.489225172s trace:
https://jaeger.dev.storj.io/trace/19740e6f10941c1
FAIL (2025-12-30 06:23:35 UTC)
orbiter warning: context deadline exceeded
Duration: 3.489247354s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
18 out of 60 is failed (threshold=3) [X..X....X.XX.X..X..X..X..XX.......X..X......X....X...X..X.X.] duration:
5.844135072s trace:
https://jaeger.dev.storj.io/trace/6be3b4cd6faf1d0d
FAIL (2025-12-30 06:23:30 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: HGGP4MNB4MILO2VLBAAIEGTCHU6NDAYJ6SWOCDZLMJLDBYKHBT3A) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: RHDGECW5DCDXQXHRP6TIYWO26IKMK2PYG4RKKLVTHR2NPAHAL2BQ) context deadline exceeded
======LOGS======
Dec 30 06:22:55.360183247 (590h53m55.231266282s) [upload=518010]: Starting upload
Dec 30 06:22:55.360241026 (590h53m55.231324061s) [upload=518010]: Done starting upload
Dec 30 06:22:55.360421000 (590h53m55.231504045s) [upload=518010]: Uploading segments...
Dec 30 06:22:55.360528134 (590h53m55.231611189s) [upload=518010]: Got next segment. Inline: false
Dec 30 06:22:55.360572388 (590h53m55.231655433s) [upload=518010]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 06:22:55.360579201 (590h53m55.231662246s) [upload=518010]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 06:22:55.607860113 (590h53m55.478943158s) [seg_pos={0 0}, upload=518010]: Begin upload segment...
Dec 30 06:22:55.607986073 (590h53m55.479069118s) [seg_pos={0 0}, upload=518010]: Done begin upload segment.
Dec 30 06:22:55.608018014 (590h53m55.479101059s) [noise=true , address=145.79.227.41:4086 , piece_id=HPGQGWATWBYIJVFZ4N442QQOXXYCLBYIXVZ445ICDLNUVDK7R4DA, seg_pos={0 0}, upload=518010]: Uploading piece...
Dec 30 06:22:55.608102093 (590h53m55.479185138s) [noise=true , address=145.79.227.25:4619 , piece_id=44T4K3OJFH2SRBOOBM4DOYDWRFYPORVJS36MX6MLOXEYTKFCUWQQ, seg_pos={0 0}, upload=518010]: Uploading piece...
Dec 30 06:22:55.608795323 (590h53m55.479878369s) [noise=true , address=145.79.227.2:4113 , piece_id=WJDN4B6OMR33HRI57OOTWPST2B7HTFQKW2GLJWSRUQP6VCJ4YOKQ, seg_pos={0 0}, upload=518010]: Uploading piece...
Dec 30 06:22:55.608824319 (590h53m55.479907364s) [noise=true , address=145.79.227.26:4654 , piece_id=QIVXLCDAFBED6JISND5CSFCMNUMK2GR2EEETEABWZAZEKCPEM23A, seg_pos={0 0}, upload=518010]: Uploading piece...
Dec 30 06:22:55.609200033 (590h53m55.480283079s) [noise=true , address=77.90.31.21:4878 , piece_id=GBH624YJVCQ76HNNGKNRYX6XC2TMJZDCV7IMMNRWFNN6R6PYEYEQ, seg_pos={0 0}, upload=518010]: Uploading piece...
Dec 30 06:22:55.609485037 (590h53m55.480568072s) [noise=true , address=45.91.116.11:4117 , piece_id=HGGP4MNB4MILO2VLBAAIEGTCHU6NDAYJ6SWOCDZLMJLDBYKHBT3A, seg_pos={0 0}, upload=518010]: Uploading piece...
Dec 30 06:22:55.609774378 (590h53m55.480857423s) [noise=true , address=145.79.227.37:4567 , piece_id=KSHILBUBXW6VSM7I6I6OFQDPEPFCXU33RB3CEDVBSULIHOUWTTQQ, seg_pos={0 0}, upload=518010]: Uploading piece...
Dec 30 06:22:55.610163528 (590h53m55.481246574s) [noise=true , address=145.79.227.28:4725 , piece_id=54JENCT2LH6LJNUFD2E7AWXVITTB5F245MJBH6UK5RVSZDF7CZHQ, seg_pos={0 0}, upload=518010]: Uploading piece...
Dec 30 06:22:55.610182295 (590h53m55.48126533s) [noise=true , address=45.91.116.13:4154 , piece_id=KGTUENYOJYHQ6QBIIIZF7OMLN4TLGLKR4MWWLOHHF4MPUYKZW5GA, seg_pos={0 0}, upload=518010]: Uploading piece...
Dec 30 06:22:55.610596634 (590h53m55.481679679s) [noise=true , address=45.91.116.14:4191 , piece_id=RHDGECW5DCDXQXHRP6TIYWO26IKMK2PYG4RKKLVTHR2NPAHAL2BQ, seg_pos={0 0}, upload=518010]: Uploading piece...
Dec 30 06:22:55.611011012 (590h53m55.482094077s) [noise=true , address=145.79.227.6:4258 , piece_id=6S55CJD4OZ3EBPBQNFXSCOY3PG52KJ5KLIKYPSUJKRXRNZV75TYA, seg_pos={0 0}, upload=518010]: Uploading piece...
Dec 30 06:22:55.611459025 (590h53m55.48254207s) [noise=true , address=145.79.227.9:4365 , piece_id=BEFHVJMJT5VWCZRFNZXOV74YOHUHW6WKXIHJJ5JHCVYWAETKLNMQ, seg_pos={0 0}, upload=518010]: Uploading piece...
Dec 30 06:22:55.611842175 (590h53m55.48292521s) [noise=true , address=145.79.227.51:4409 , piece_id=I3EPPRI7BZYTP2BMOOS3WEW33JEJQ3R7TSP6M6G66TTZ3VERIWIA, seg_pos={0 0}, upload=518010]: Uploading piece...
Dec 30 06:22:55.612157736 (590h53m55.483240781s) [noise=true , address=77.90.31.19:4842 , piece_id=4Q3LMTNN373ZBS6WTWSSASDMXY2YT47E73WC5HVBK4GXNPQEZ5PQ, seg_pos={0 0}, upload=518010]: Uploading piece...
Dec 30 06:22:55.612953160 (590h53m55.484036205s) [noise=true , address=145.79.227.7:4294 , piece_id=SO2FCMQ2LB5LE3LH5BNSRFOUBFMNXWHTTK4VQTMTVJQTPIEF4SJA, seg_pos={0 0}, upload=518010]: Uploading piece...
Dec 30 06:22:55.613081705 (590h53m55.48416475s) [noise=true , address=145.79.227.31:4798 , piece_id=WC3J7TLWHZC4XYMT2GVAY64PDQ6YYTNQ3RT2C4RF7TIK75FIKPNA, seg_pos={0 0}, upload=518010]: Uploading piece...
Dec 30 06:22:57.216081529 (590h53m57.087164574s) [upload=518010]: Next returned nil segment
Dec 30 06:22:57.216095807 (590h53m57.087178842s) [upload=518010]: Waiting for error group managing segments...
Dec 30 06:22:57.597153748 (590h53m57.468236783s) [noise=true , address=145.79.227.25:4619 , piece_id=44T4K3OJFH2SRBOOBM4DOYDWRFYPORVJS36MX6MLOXEYTKFCUWQQ, seg_pos={0 0}, upload=518010]: Done uploading piece. err: <nil>
Dec 30 06:22:57.603011944 (590h53m57.474094979s) [noise=true , address=145.79.227.41:4086 , piece_id=HPGQGWATWBYIJVFZ4N442QQOXXYCLBYIXVZ445ICDLNUVDK7R4DA, seg_pos={0 0}, upload=518010]: Done uploading piece. err: <nil>
Dec 30 06:22:57.604362786 (590h53m57.475445821s) [noise=true , address=145.79.227.26:4654 , piece_id=QIVXLCDAFBED6JISND5CSFCMNUMK2GR2EEETEABWZAZEKCPEM23A, seg_pos={0 0}, upload=518010]: Done uploading piece. err: <nil>
Dec 30 06:22:57.604637218 (590h53m57.475720263s) [noise=true , address=145.79.227.28:4725 , piece_id=54JENCT2LH6LJNUFD2E7AWXVITTB5F245MJBH6UK5RVSZDF7CZHQ, seg_pos={0 0}, upload=518010]: Done uploading piece. err: <nil>
Dec 30 06:22:57.608254746 (590h53m57.479337792s) [noise=true , address=145.79.227.7:4294 , piece_id=SO2FCMQ2LB5LE3LH5BNSRFOUBFMNXWHTTK4VQTMTVJQTPIEF4SJA, seg_pos={0 0}, upload=518010]: Done uploading piece. err: <nil>
Dec 30 06:22:57.610501584 (590h53m57.481584649s) [noise=true , address=145.79.227.31:4798 , piece_id=WC3J7TLWHZC4XYMT2GVAY64PDQ6YYTNQ3RT2C4RF7TIK75FIKPNA, seg_pos={0 0}, upload=518010]: Done uploading piece. err: <nil>
Dec 30 06:22:57.612998329 (590h53m57.484081364s) [noise=true , address=145.79.227.6:4258 , piece_id=6S55CJD4OZ3EBPBQNFXSCOY3PG52KJ5KLIKYPSUJKRXRNZV75TYA, seg_pos={0 0}, upload=518010]: Done uploading piece. err: <nil>
Dec 30 06:22:57.613338527 (590h53m57.484421572s) [noise=true , address=145.79.227.2:4113 , piece_id=WJDN4B6OMR33HRI57OOTWPST2B7HTFQKW2GLJWSRUQP6VCJ4YOKQ, seg_pos={0 0}, upload=518010]: Done uploading piece. err: <nil>
Dec 30 06:22:57.616486691 (590h53m57.487569726s) [noise=true , address=145.79.227.51:4409 , piece_id=I3EPPRI7BZYTP2BMOOS3WEW33JEJQ3R7TSP6M6G66TTZ3VERIWIA, seg_pos={0 0}, upload=518010]: Done uploading piece. err: <nil>
Dec 30 06:22:57.618355390 (590h53m57.489438435s) [noise=true , address=145.79.227.9:4365 , piece_id=BEFHVJMJT5VWCZRFNZXOV74YOHUHW6WKXIHJJ5JHCVYWAETKLNMQ, seg_pos={0 0}, upload=518010]: Done uploading piece. err: <nil>
Dec 30 06:22:57.633964552 (590h53m57.505047587s) [noise=true , address=145.79.227.37:4567 , piece_id=KSHILBUBXW6VSM7I6I6OFQDPEPFCXU33RB3CEDVBSULIHOUWTTQQ, seg_pos={0 0}, upload=518010]: Done uploading piece. err: <nil>
Dec 30 06:22:57.916268732 (590h53m57.787351777s) [noise=true , address=45.91.116.14:4191 , piece_id=RHDGECW5DCDXQXHRP6TIYWO26IKMK2PYG4RKKLVTHR2NPAHAL2BQ, seg_pos={0 0}, upload=518010]: Done uploading piece. err: <nil>
Dec 30 06:22:57.936750212 (590h53m57.807833257s) [noise=true , address=45.91.116.11:4117 , piece_id=HGGP4MNB4MILO2VLBAAIEGTCHU6NDAYJ6SWOCDZLMJLDBYKHBT3A, seg_pos={0 0}, upload=518010]: Done uploading piece. err: <nil>
Dec 30 06:22:57.936770772 (590h53m57.807853807s) [seg_pos={0 0}, upload=518010]: Segment reached optimal threshold of 13 pieces.
Dec 30 06:22:57.936992333 (590h53m57.808075368s) [noise=true , address=45.91.116.13:4154 , piece_id=KGTUENYOJYHQ6QBIIIZF7OMLN4TLGLKR4MWWLOHHF4MPUYKZW5GA, seg_pos={0 0}, upload=518010]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:57.936999947 (590h53m57.808082992s) [noise=true , address=45.91.116.13:4154 , piece_id=KGTUENYOJYHQ6QBIIIZF7OMLN4TLGLKR4MWWLOHHF4MPUYKZW5GA, seg_pos={0 0}, upload=518010]: Long tail cancellation due to optimal threshold
Dec 30 06:22:57.937388457 (590h53m57.808471502s) [noise=true , address=77.90.31.21:4878 , piece_id=GBH624YJVCQ76HNNGKNRYX6XC2TMJZDCV7IMMNRWFNN6R6PYEYEQ, seg_pos={0 0}, upload=518010]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: read tcp 188.166.225.170:35422->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 30 06:22:57.937401142 (590h53m57.808484177s) [noise=true , address=77.90.31.21:4878 , piece_id=GBH624YJVCQ76HNNGKNRYX6XC2TMJZDCV7IMMNRWFNN6R6PYEYEQ, seg_pos={0 0}, upload=518010]: Long tail cancellation due to optimal threshold
Dec 30 06:22:57.937468039 (590h53m57.808551074s) [noise=true , address=77.90.31.19:4842 , piece_id=4Q3LMTNN373ZBS6WTWSSASDMXY2YT47E73WC5HVBK4GXNPQEZ5PQ, seg_pos={0 0}, upload=518010]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:57.937476044 (590h53m57.808559079s) [noise=true , address=77.90.31.19:4842 , piece_id=4Q3LMTNN373ZBS6WTWSSASDMXY2YT47E73WC5HVBK4GXNPQEZ5PQ, seg_pos={0 0}, upload=518010]: Long tail cancellation due to optimal threshold
Dec 30 06:22:57.937536089 (590h53m57.808619134s) [seg_pos={0 0}, upload=518010]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 06:22:57.937604439 (590h53m57.808687484s) [upload=518010]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:22:57.937610150 (590h53m57.808693195s) [upload=518010]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:22:58.225647716 (590h53m58.096730751s) [upload=518010]: Done uploading segments...
Dec 30 06:22:58.225665971 (590h53m58.096749006s) [upload=518010]: Upload finished. err: <nil>
Duration: 5.844149629s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
18 out of 60 is failed (threshold=3) [.X..X....X.XX.X..X..X..X..XX.......X..X......X....X...X..X.X] duration:
30.002955209s trace:
https://jaeger.dev.storj.io/trace/6be3b4cd6faf1d0d
FAIL (2025-12-30 06:23:24 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: HGGP4MNB4MILO2VLBAAIEGTCHU6NDAYJ6SWOCDZLMJLDBYKHBT3A) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: RHDGECW5DCDXQXHRP6TIYWO26IKMK2PYG4RKKLVTHR2NPAHAL2BQ) context deadline exceeded
======LOGS======
Dec 30 06:22:55.360183247 (590h53m55.231266282s) [upload=518010]: Starting upload
Dec 30 06:22:55.360241026 (590h53m55.231324061s) [upload=518010]: Done starting upload
Dec 30 06:22:55.360421000 (590h53m55.231504045s) [upload=518010]: Uploading segments...
Dec 30 06:22:55.360528134 (590h53m55.231611189s) [upload=518010]: Got next segment. Inline: false
Dec 30 06:22:55.360572388 (590h53m55.231655433s) [upload=518010]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 06:22:55.360579201 (590h53m55.231662246s) [upload=518010]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 06:22:55.607860113 (590h53m55.478943158s) [seg_pos={0 0}, upload=518010]: Begin upload segment...
Dec 30 06:22:55.607986073 (590h53m55.479069118s) [seg_pos={0 0}, upload=518010]: Done begin upload segment.
Dec 30 06:22:55.608018014 (590h53m55.479101059s) [noise=true , address=145.79.227.41:4086 , piece_id=HPGQGWATWBYIJVFZ4N442QQOXXYCLBYIXVZ445ICDLNUVDK7R4DA, seg_pos={0 0}, upload=518010]: Uploading piece...
Dec 30 06:22:55.608102093 (590h53m55.479185138s) [noise=true , address=145.79.227.25:4619 , piece_id=44T4K3OJFH2SRBOOBM4DOYDWRFYPORVJS36MX6MLOXEYTKFCUWQQ, seg_pos={0 0}, upload=518010]: Uploading piece...
Dec 30 06:22:55.608795323 (590h53m55.479878369s) [noise=true , address=145.79.227.2:4113 , piece_id=WJDN4B6OMR33HRI57OOTWPST2B7HTFQKW2GLJWSRUQP6VCJ4YOKQ, seg_pos={0 0}, upload=518010]: Uploading piece...
Dec 30 06:22:55.608824319 (590h53m55.479907364s) [noise=true , address=145.79.227.26:4654 , piece_id=QIVXLCDAFBED6JISND5CSFCMNUMK2GR2EEETEABWZAZEKCPEM23A, seg_pos={0 0}, upload=518010]: Uploading piece...
Dec 30 06:22:55.609200033 (590h53m55.480283079s) [noise=true , address=77.90.31.21:4878 , piece_id=GBH624YJVCQ76HNNGKNRYX6XC2TMJZDCV7IMMNRWFNN6R6PYEYEQ, seg_pos={0 0}, upload=518010]: Uploading piece...
Dec 30 06:22:55.609485037 (590h53m55.480568072s) [noise=true , address=45.91.116.11:4117 , piece_id=HGGP4MNB4MILO2VLBAAIEGTCHU6NDAYJ6SWOCDZLMJLDBYKHBT3A, seg_pos={0 0}, upload=518010]: Uploading piece...
Dec 30 06:22:55.609774378 (590h53m55.480857423s) [noise=true , address=145.79.227.37:4567 , piece_id=KSHILBUBXW6VSM7I6I6OFQDPEPFCXU33RB3CEDVBSULIHOUWTTQQ, seg_pos={0 0}, upload=518010]: Uploading piece...
Dec 30 06:22:55.610163528 (590h53m55.481246574s) [noise=true , address=145.79.227.28:4725 , piece_id=54JENCT2LH6LJNUFD2E7AWXVITTB5F245MJBH6UK5RVSZDF7CZHQ, seg_pos={0 0}, upload=518010]: Uploading piece...
Dec 30 06:22:55.610182295 (590h53m55.48126533s) [noise=true , address=45.91.116.13:4154 , piece_id=KGTUENYOJYHQ6QBIIIZF7OMLN4TLGLKR4MWWLOHHF4MPUYKZW5GA, seg_pos={0 0}, upload=518010]: Uploading piece...
Dec 30 06:22:55.610596634 (590h53m55.481679679s) [noise=true , address=45.91.116.14:4191 , piece_id=RHDGECW5DCDXQXHRP6TIYWO26IKMK2PYG4RKKLVTHR2NPAHAL2BQ, seg_pos={0 0}, upload=518010]: Uploading piece...
Dec 30 06:22:55.611011012 (590h53m55.482094077s) [noise=true , address=145.79.227.6:4258 , piece_id=6S55CJD4OZ3EBPBQNFXSCOY3PG52KJ5KLIKYPSUJKRXRNZV75TYA, seg_pos={0 0}, upload=518010]: Uploading piece...
Dec 30 06:22:55.611459025 (590h53m55.48254207s) [noise=true , address=145.79.227.9:4365 , piece_id=BEFHVJMJT5VWCZRFNZXOV74YOHUHW6WKXIHJJ5JHCVYWAETKLNMQ, seg_pos={0 0}, upload=518010]: Uploading piece...
Dec 30 06:22:55.611842175 (590h53m55.48292521s) [noise=true , address=145.79.227.51:4409 , piece_id=I3EPPRI7BZYTP2BMOOS3WEW33JEJQ3R7TSP6M6G66TTZ3VERIWIA, seg_pos={0 0}, upload=518010]: Uploading piece...
Dec 30 06:22:55.612157736 (590h53m55.483240781s) [noise=true , address=77.90.31.19:4842 , piece_id=4Q3LMTNN373ZBS6WTWSSASDMXY2YT47E73WC5HVBK4GXNPQEZ5PQ, seg_pos={0 0}, upload=518010]: Uploading piece...
Dec 30 06:22:55.612953160 (590h53m55.484036205s) [noise=true , address=145.79.227.7:4294 , piece_id=SO2FCMQ2LB5LE3LH5BNSRFOUBFMNXWHTTK4VQTMTVJQTPIEF4SJA, seg_pos={0 0}, upload=518010]: Uploading piece...
Dec 30 06:22:55.613081705 (590h53m55.48416475s) [noise=true , address=145.79.227.31:4798 , piece_id=WC3J7TLWHZC4XYMT2GVAY64PDQ6YYTNQ3RT2C4RF7TIK75FIKPNA, seg_pos={0 0}, upload=518010]: Uploading piece...
Dec 30 06:22:57.216081529 (590h53m57.087164574s) [upload=518010]: Next returned nil segment
Dec 30 06:22:57.216095807 (590h53m57.087178842s) [upload=518010]: Waiting for error group managing segments...
Dec 30 06:22:57.597153748 (590h53m57.468236783s) [noise=true , address=145.79.227.25:4619 , piece_id=44T4K3OJFH2SRBOOBM4DOYDWRFYPORVJS36MX6MLOXEYTKFCUWQQ, seg_pos={0 0}, upload=518010]: Done uploading piece. err: <nil>
Dec 30 06:22:57.603011944 (590h53m57.474094979s) [noise=true , address=145.79.227.41:4086 , piece_id=HPGQGWATWBYIJVFZ4N442QQOXXYCLBYIXVZ445ICDLNUVDK7R4DA, seg_pos={0 0}, upload=518010]: Done uploading piece. err: <nil>
Dec 30 06:22:57.604362786 (590h53m57.475445821s) [noise=true , address=145.79.227.26:4654 , piece_id=QIVXLCDAFBED6JISND5CSFCMNUMK2GR2EEETEABWZAZEKCPEM23A, seg_pos={0 0}, upload=518010]: Done uploading piece. err: <nil>
Dec 30 06:22:57.604637218 (590h53m57.475720263s) [noise=true , address=145.79.227.28:4725 , piece_id=54JENCT2LH6LJNUFD2E7AWXVITTB5F245MJBH6UK5RVSZDF7CZHQ, seg_pos={0 0}, upload=518010]: Done uploading piece. err: <nil>
Dec 30 06:22:57.608254746 (590h53m57.479337792s) [noise=true , address=145.79.227.7:4294 , piece_id=SO2FCMQ2LB5LE3LH5BNSRFOUBFMNXWHTTK4VQTMTVJQTPIEF4SJA, seg_pos={0 0}, upload=518010]: Done uploading piece. err: <nil>
Dec 30 06:22:57.610501584 (590h53m57.481584649s) [noise=true , address=145.79.227.31:4798 , piece_id=WC3J7TLWHZC4XYMT2GVAY64PDQ6YYTNQ3RT2C4RF7TIK75FIKPNA, seg_pos={0 0}, upload=518010]: Done uploading piece. err: <nil>
Dec 30 06:22:57.612998329 (590h53m57.484081364s) [noise=true , address=145.79.227.6:4258 , piece_id=6S55CJD4OZ3EBPBQNFXSCOY3PG52KJ5KLIKYPSUJKRXRNZV75TYA, seg_pos={0 0}, upload=518010]: Done uploading piece. err: <nil>
Dec 30 06:22:57.613338527 (590h53m57.484421572s) [noise=true , address=145.79.227.2:4113 , piece_id=WJDN4B6OMR33HRI57OOTWPST2B7HTFQKW2GLJWSRUQP6VCJ4YOKQ, seg_pos={0 0}, upload=518010]: Done uploading piece. err: <nil>
Dec 30 06:22:57.616486691 (590h53m57.487569726s) [noise=true , address=145.79.227.51:4409 , piece_id=I3EPPRI7BZYTP2BMOOS3WEW33JEJQ3R7TSP6M6G66TTZ3VERIWIA, seg_pos={0 0}, upload=518010]: Done uploading piece. err: <nil>
Dec 30 06:22:57.618355390 (590h53m57.489438435s) [noise=true , address=145.79.227.9:4365 , piece_id=BEFHVJMJT5VWCZRFNZXOV74YOHUHW6WKXIHJJ5JHCVYWAETKLNMQ, seg_pos={0 0}, upload=518010]: Done uploading piece. err: <nil>
Dec 30 06:22:57.633964552 (590h53m57.505047587s) [noise=true , address=145.79.227.37:4567 , piece_id=KSHILBUBXW6VSM7I6I6OFQDPEPFCXU33RB3CEDVBSULIHOUWTTQQ, seg_pos={0 0}, upload=518010]: Done uploading piece. err: <nil>
Dec 30 06:22:57.916268732 (590h53m57.787351777s) [noise=true , address=45.91.116.14:4191 , piece_id=RHDGECW5DCDXQXHRP6TIYWO26IKMK2PYG4RKKLVTHR2NPAHAL2BQ, seg_pos={0 0}, upload=518010]: Done uploading piece. err: <nil>
Dec 30 06:22:57.936750212 (590h53m57.807833257s) [noise=true , address=45.91.116.11:4117 , piece_id=HGGP4MNB4MILO2VLBAAIEGTCHU6NDAYJ6SWOCDZLMJLDBYKHBT3A, seg_pos={0 0}, upload=518010]: Done uploading piece. err: <nil>
Dec 30 06:22:57.936770772 (590h53m57.807853807s) [seg_pos={0 0}, upload=518010]: Segment reached optimal threshold of 13 pieces.
Dec 30 06:22:57.936992333 (590h53m57.808075368s) [noise=true , address=45.91.116.13:4154 , piece_id=KGTUENYOJYHQ6QBIIIZF7OMLN4TLGLKR4MWWLOHHF4MPUYKZW5GA, seg_pos={0 0}, upload=518010]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:57.936999947 (590h53m57.808082992s) [noise=true , address=45.91.116.13:4154 , piece_id=KGTUENYOJYHQ6QBIIIZF7OMLN4TLGLKR4MWWLOHHF4MPUYKZW5GA, seg_pos={0 0}, upload=518010]: Long tail cancellation due to optimal threshold
Dec 30 06:22:57.937388457 (590h53m57.808471502s) [noise=true , address=77.90.31.21:4878 , piece_id=GBH624YJVCQ76HNNGKNRYX6XC2TMJZDCV7IMMNRWFNN6R6PYEYEQ, seg_pos={0 0}, upload=518010]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: read tcp 188.166.225.170:35422->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 30 06:22:57.937401142 (590h53m57.808484177s) [noise=true , address=77.90.31.21:4878 , piece_id=GBH624YJVCQ76HNNGKNRYX6XC2TMJZDCV7IMMNRWFNN6R6PYEYEQ, seg_pos={0 0}, upload=518010]: Long tail cancellation due to optimal threshold
Dec 30 06:22:57.937468039 (590h53m57.808551074s) [noise=true , address=77.90.31.19:4842 , piece_id=4Q3LMTNN373ZBS6WTWSSASDMXY2YT47E73WC5HVBK4GXNPQEZ5PQ, seg_pos={0 0}, upload=518010]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:57.937476044 (590h53m57.808559079s) [noise=true , address=77.90.31.19:4842 , piece_id=4Q3LMTNN373ZBS6WTWSSASDMXY2YT47E73WC5HVBK4GXNPQEZ5PQ, seg_pos={0 0}, upload=518010]: Long tail cancellation due to optimal threshold
Dec 30 06:22:57.937536089 (590h53m57.808619134s) [seg_pos={0 0}, upload=518010]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 06:22:57.937604439 (590h53m57.808687484s) [upload=518010]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:22:57.937610150 (590h53m57.808693195s) [upload=518010]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:22:58.225647716 (590h53m58.096730751s) [upload=518010]: Done uploading segments...
Dec 30 06:22:58.225665971 (590h53m58.096749006s) [upload=518010]: Upload finished. err: <nil>
Duration: 30.00303442s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..................................................X.........] duration:
12.583937189s trace:
https://jaeger.dev.storj.io/trace/deb96436003dee
FAIL (2025-12-30 06:23:16 UTC)
orbiter warning: read tcp 188.166.225.170:52458->45.38.24.2:443: read: connection reset by peer
Duration: 12.583968138s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [..................................X...........X.............] duration:
4.695482413s trace:
https://jaeger.dev.storj.io/trace/19740e6f10941c1
FAIL (2025-12-30 06:23:15 UTC)
orbiter warning: context deadline exceeded
Duration: 4.695678827s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [...................................X...X......X............X] duration:
1m0.003168075s trace:
https://jaeger.dev.storj.io/trace/45b769fc71e13448
FAIL (2025-12-30 06:23:10 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
.577697683 (590h53m11.448780728s) [noise=true , address=193.149.164.143:4704 , piece_id=4T2IJIA6WHI5LMIUP5BLJLCD3GRY3A6PXR6G5OWJWUSYZT5ELGHQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578173528 (590h53m11.449256573s) [noise=true , address=64.49.13.27:4965 , piece_id=CO4O5F2IBZAAKBZBV3HWNNOXZOGLNNDPALWMQZ3DEM3MREASVFEA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578542992 (590h53m11.449626027s) [noise=true , address=77.90.31.15:4729 , piece_id=NVMXFEILHGT622IIPYYT4FFHC2WZLHMCZM4VT5WUPHRAPH3DX6TA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578846911 (590h53m11.449929946s) [noise=true , address=145.79.227.43:4177 , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579458044 (590h53m11.450541089s) [noise=true , address=45.91.116.11:4150 , piece_id=E66IDHTJDLYLYJIOWMOIKOOWLXCUNCMBIQ7SQBAD3VTJF6VX5JLQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579891209 (590h53m11.450974245s) [noise=true , address=145.79.227.29:4945 , piece_id=SHL73IDEI54GRCWYNWHRUSJZZ4UUXREAX54DFJLTG5QPZCF5QKIQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579915535 (590h53m11.45099858s) [noise=true , address=145.79.227.30:4772 , piece_id=PVMD5DWRRPVARTSZ6O7HN6MUVTZUYWLYSNQR5GDMNKUJZGXMYPFQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580325466 (590h53m11.451408511s) [noise=true , address=77.90.31.12:4603 , piece_id=2I4L7F2WEGK5OSLODJSJ4672URBA7DUGXWZKTYB6ZNA2TQIGVXWA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580379599 (590h53m11.451462644s) [noise=true , address=77.90.31.14:4671 , piece_id=263TRISJ5ADFTX43ULSSGBAF2KMOXAPISJ56LLE53D74XOEMQ5EA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580692044 (590h53m11.451775089s) [noise=true , address=145.79.227.3:4175 , piece_id=UCV5WQNVWEAMEDBWDNXEVSQEGXPJE7KD6YSWC5A4WXYUKUCK4NLA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580857859 (590h53m11.451940894s) [noise=true , address=77.90.31.19:4860 , piece_id=U2MNFOAD7FHCDBE4FUYOROIP3P33ROJ7K75SF5Z747EFKFTF7RGA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581089210 (590h53m11.452172255s) [noise=true , address=145.79.227.44:4203 , piece_id=WHCKQ2FRLYAZE7Q6HYZNTS4EISVHZNAPDDKH34WN33ALK4MVK2EA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581351320 (590h53m11.452434355s) [noise=true , address=64.49.13.154:4928 , piece_id=SEVBZBMNGURP5TRUEBA4PHRSA6MNANVJUKINALJUWPQWBGKVYNTA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581457091 (590h53m11.452540126s) [noise=true , address=145.79.227.34:4909 , piece_id=3QSWTNR5LGUUAY3N5HJKSPL3TRQ6WO5F5OD4VTXKWHWUXUF7PVVQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581863144 (590h53m11.452946179s) [noise=true , address=145.79.227.33:4881 , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.584010443 (590h53m11.455093478s) [noise=true , address=145.79.227.40:4070 , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.585122791 (590h53m11.456205826s) [noise=true , address=45.91.116.14:4217 , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:12.983305402 (590h53m12.854388447s) [upload=517998]: Next returned nil segment
Dec 30 06:22:12.983321313 (590h53m12.854404348s) [upload=517998]: Waiting for error group managing segments...
Dec 30 06:22:13.379703053 (590h53m13.250786088s) [noise=true , address=64.49.13.27:4965 , piece_id=CO4O5F2IBZAAKBZBV3HWNNOXZOGLNNDPALWMQZ3DEM3MREASVFEA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.418055622 (590h53m13.289138657s) [noise=true , address=145.79.227.3:4175 , piece_id=UCV5WQNVWEAMEDBWDNXEVSQEGXPJE7KD6YSWC5A4WXYUKUCK4NLA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.509034160 (590h53m13.380117205s) [noise=true , address=64.49.13.154:4928 , piece_id=SEVBZBMNGURP5TRUEBA4PHRSA6MNANVJUKINALJUWPQWBGKVYNTA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.642886154 (590h53m13.513969199s) [noise=true , address=77.90.31.14:4671 , piece_id=263TRISJ5ADFTX43ULSSGBAF2KMOXAPISJ56LLE53D74XOEMQ5EA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.788145520 (590h53m13.659228555s) [noise=true , address=77.90.31.12:4603 , piece_id=2I4L7F2WEGK5OSLODJSJ4672URBA7DUGXWZKTYB6ZNA2TQIGVXWA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.858878757 (590h53m13.729961792s) [noise=true , address=145.79.227.49:4369 , piece_id=XUEXGNJJIJ4DUBDNUV7CKF6J5JYHSLZAEPVGEDGBUBY42BIP4XEQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.879550569 (590h53m13.750633614s) [noise=true , address=145.79.227.34:4909 , piece_id=3QSWTNR5LGUUAY3N5HJKSPL3TRQ6WO5F5OD4VTXKWHWUXUF7PVVQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.940897723 (590h53m13.811980769s) [noise=true , address=77.90.31.19:4860 , piece_id=U2MNFOAD7FHCDBE4FUYOROIP3P33ROJ7K75SF5Z747EFKFTF7RGA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.989791936 (590h53m13.860874981s) [noise=true , address=45.91.116.11:4150 , piece_id=E66IDHTJDLYLYJIOWMOIKOOWLXCUNCMBIQ7SQBAD3VTJF6VX5JLQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.289516718 (590h53m14.160599753s) [noise=true , address=145.79.227.44:4203 , piece_id=WHCKQ2FRLYAZE7Q6HYZNTS4EISVHZNAPDDKH34WN33ALK4MVK2EA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.338651537 (590h53m14.209734582s) [noise=true , address=77.90.31.15:4729 , piece_id=NVMXFEILHGT622IIPYYT4FFHC2WZLHMCZM4VT5WUPHRAPH3DX6TA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.344145168 (590h53m14.215228213s) [noise=true , address=145.79.227.30:4772 , piece_id=PVMD5DWRRPVARTSZ6O7HN6MUVTZUYWLYSNQR5GDMNKUJZGXMYPFQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.667232847 (590h53m14.538315892s) [noise=true , address=193.149.164.143:4704 , piece_id=4T2IJIA6WHI5LMIUP5BLJLCD3GRY3A6PXR6G5OWJWUSYZT5ELGHQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.671388990 (590h53m14.542472025s) [noise=true , address=145.79.227.29:4945 , piece_id=SHL73IDEI54GRCWYNWHRUSJZZ4UUXREAX54DFJLTG5QPZCF5QKIQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.671415190 (590h53m14.542498226s) [seg_pos={0 0}, upload=517998]: Segment reached optimal threshold of 14 pieces.
Dec 30 06:22:14.671803600 (590h53m14.542886655s) [noise=true , address=145.79.227.40:4070 , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hU1LwL2gX7rVnrNgzaBCYB37ziqf7s7aUgtF2Uc3antReQ6me): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.671814470 (590h53m14.542897525s) [noise=true , address=145.79.227.40:4070 , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672014932 (590h53m14.543097977s) [noise=true , address=145.79.227.43:4177 , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bhN3cR22QuAYihnnB3x3R97UZRWJ8sFPP8fjQcXZ7qhNZphZG): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672022637 (590h53m14.543105672s) [noise=true , address=145.79.227.43:4177 , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672088933 (590h53m14.543171988s) [noise=true , address=145.79.227.33:4881 , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672127677 (590h53m14.543210742s) [noise=true , address=145.79.227.33:4881 , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672312739 (590h53m14.543395774s) [noise=true , address=45.91.116.14:4217 , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xSXWFiq27Dd2vBDD9vVnhDHVG59UmDKn3hVAhf3oyo5goVNaE): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672319953 (590h53m14.543402988s) [noise=true , address=45.91.116.14:4217 , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672337847 (590h53m14.543420882s) [seg_pos={0 0}, upload=517998]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 06:22:14.672452947 (590h53m14.543535982s) [upload=517998]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:22:14.672458507 (590h53m14.543541542s) [upload=517998]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:22:14.981437069 (590h53m14.852520114s) [upload=517998]: Done uploading segments...
Dec 30 06:22:14.981456546 (590h53m14.852539581s) [upload=517998]: Upload finished. err: <nil>
Duration: 1m0.003188473s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [...................................X...........X............] duration:
3.510535s trace:
https://jaeger.dev.storj.io/trace/19740e6f10941c1
FAIL (2025-12-30 06:22:50 UTC)
orbiter warning: context deadline exceeded
Duration: 3.510560418s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...................................................X........] duration:
10.535045915s trace:
https://jaeger.dev.storj.io/trace/deb96436003dee
FAIL (2025-12-30 06:22:36 UTC)
orbiter warning: read tcp 188.166.225.170:52458->45.38.24.2:443: read: connection reset by peer
Duration: 10.535079699s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
17 out of 60 is failed (threshold=3) [..X..X....X.XX.X..X..X..X..XX.......X..X......X....X...X..X.] duration:
5.984835114s trace:
https://jaeger.dev.storj.io/trace/65e658867ff66eaf
FAIL (2025-12-30 06:22:16 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: TVFZBRPDXCKHVGMBKFOOMWONAZK5KJG7DLJ62FNQXM7T5P5AHB3Q) context deadline exceeded; piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: U6HUZIN25WB2CCPJ54QEKGUSFZ4KWE6FJ2EDLNT2VQMY2HJK3TCQ) context deadline exceeded
======LOGS======
30 06:21:26.267761284 (590h52m26.138844319s) [upload=517989]: Starting upload
Dec 30 06:21:26.267835385 (590h52m26.13891844s) [upload=517989]: Done starting upload
Dec 30 06:21:26.267927901 (590h52m26.139010936s) [upload=517989]: Uploading segments...
Dec 30 06:21:26.268098546 (590h52m26.139181591s) [upload=517989]: Got next segment. Inline: false
Dec 30 06:21:26.268143722 (590h52m26.139226757s) [upload=517989]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 06:21:26.268152128 (590h52m26.139235163s) [upload=517989]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 06:21:26.515138137 (590h52m26.386221182s) [seg_pos={0 0}, upload=517989]: Begin upload segment...
Dec 30 06:21:26.515190235 (590h52m26.386273281s) [seg_pos={0 0}, upload=517989]: Done begin upload segment.
Dec 30 06:21:26.515226214 (590h52m26.386309259s) [noise=true , address=45.91.116.7:4009 , piece_id=IIOD3ELLRYAL7JXXADWX6TU4KPDA55ZH543CBCSVZYOLMFS2LPVA, seg_pos={0 0}, upload=517989]: Uploading piece...
Dec 30 06:21:26.515415906 (590h52m26.386498951s) [noise=true , address=145.79.227.15:4876 , piece_id=VP27XUZSPN3CWTWSK7Z67ESPR4K6QG2IZ3F44NRRFVVIE4SYWM2A, seg_pos={0 0}, upload=517989]: Uploading piece...
Dec 30 06:21:26.515955713 (590h52m26.387038748s) [noise=true , address=145.79.227.10:4402 , piece_id=B3QDVFATNS3ELKAPMJMPL7VQ7REUZX75N35T3UFPTAJTIKSMY3VA, seg_pos={0 0}, upload=517989]: Uploading piece...
Dec 30 06:21:26.516497335 (590h52m26.38758038s) [noise=true , address=145.79.227.6:4259 , piece_id=2H3JKA3KDOSALYHMV3TENUXY3SXD4EH5IXHETMRLO572FSCLMYMA, seg_pos={0 0}, upload=517989]: Uploading piece...
Dec 30 06:21:26.516929918 (590h52m26.388012963s) [noise=true , address=145.79.227.34:4907 , piece_id=L76LLBAPKNPYEHZWXRTOECRHRUU6VMEDCEWCIYVULFTUWGJCG5QA, seg_pos={0 0}, upload=517989]: Uploading piece...
Dec 30 06:21:26.517317215 (590h52m26.388400251s) [noise=true , address=145.79.227.26:4654 , piece_id=MCTENZWHS5NYN3KQMANVEZKXN5KGTJDDT552IFI65IUU24DHTVLA, seg_pos={0 0}, upload=517989]: Uploading piece...
Dec 30 06:21:26.517684736 (590h52m26.388767771s) [noise=true , address=145.79.227.47:4266 , piece_id=SF4O5OK7WLKI76F2AAP5QT4KH2GZ2L2CQP5HKHIS7SMU7S2BOEIQ, seg_pos={0 0}, upload=517989]: Uploading piece...
Dec 30 06:21:26.517703962 (590h52m26.388787007s) [noise=true , address=77.90.31.15:4697 , piece_id=U6HUZIN25WB2CCPJ54QEKGUSFZ4KWE6FJ2EDLNT2VQMY2HJK3TCQ, seg_pos={0 0}, upload=517989]: Uploading piece...
Dec 30 06:21:26.518244662 (590h52m26.389327707s) [noise=true , address=145.79.227.7:4294 , piece_id=L4ENKEYIDICT4SW7QTXB7IXSTXQ5XNK2HJQKCH7CVWD454BRND6Q, seg_pos={0 0}, upload=517989]: Uploading piece...
Dec 30 06:21:26.518310287 (590h52m26.389393322s) [noise=true , address=145.79.227.50:4373 , piece_id=CHF3QXLYY3OPJHPHFHUEX4O4KSKE625O7BUJ7VZYSDTOHJRXRFPQ, seg_pos={0 0}, upload=517989]: Uploading piece...
Dec 30 06:21:26.518649933 (590h52m26.389732978s) [noise=true , address=145.79.227.18:4006 , piece_id=JYGYYNDOQ6NCL6E3GECNBTUJ4V2LIDY5QGINEAVCZTAKO4LH66VA, seg_pos={0 0}, upload=517989]: Uploading piece...
Dec 30 06:21:26.518857398 (590h52m26.389940443s) [noise=true , address=145.79.227.8:4329 , piece_id=ZXCFNCSW5H7ZPGBFC6LHX3RT5NZ6YHOTSDWV3GQVCYYQ7I57VADA, seg_pos={0 0}, upload=517989]: Uploading piece...
Dec 30 06:21:26.519034176 (590h52m26.390117221s) [noise=true , address=145.79.227.20:4439 , piece_id=VANS6EWTKHBYGVKVIHYIVJKQBUVLSDOEV2VQ6TD4NEHTF5SJPQVA, seg_pos={0 0}, upload=517989]: Uploading piece...
Dec 30 06:21:26.519374443 (590h52m26.390457478s) [noise=true , address=45.91.116.5:4300 , piece_id=OKCDW5TU2YBOWJO6FGB4K6Q5DWTLPP2ZP257BORAHO5CZK27ODEQ, seg_pos={0 0}, upload=517989]: Uploading piece...
Dec 30 06:21:26.519907659 (590h52m26.390990704s) [noise=true , address=77.90.31.13:4627 , piece_id=TVFZBRPDXCKHVGMBKFOOMWONAZK5KJG7DLJ62FNQXM7T5P5AHB3Q, seg_pos={0 0}, upload=517989]: Uploading piece...
Dec 30 06:21:26.520439221 (590h52m26.391522256s) [noise=true , address=45.91.116.8:4047 , piece_id=GSG2ZHOR56FAWWZAH5EUHUNQ3SS6IHJ6ZZMVZHM2HQFN3XYTYYCA, seg_pos={0 0}, upload=517989]: Uploading piece...
Dec 30 06:21:28.214165843 (590h52m28.085248878s) [upload=517989]: Next returned nil segment
Dec 30 06:21:28.214179650 (590h52m28.085262685s) [upload=517989]: Waiting for error group managing segments...
Dec 30 06:21:28.498112992 (590h52m28.369196037s) [noise=true , address=145.79.227.34:4907 , piece_id=L76LLBAPKNPYEHZWXRTOECRHRUU6VMEDCEWCIYVULFTUWGJCG5QA, seg_pos={0 0}, upload=517989]: Done uploading piece. err: <nil>
Dec 30 06:21:28.505291993 (590h52m28.376375038s) [noise=true , address=145.79.227.15:4876 , piece_id=VP27XUZSPN3CWTWSK7Z67ESPR4K6QG2IZ3F44NRRFVVIE4SYWM2A, seg_pos={0 0}, upload=517989]: Done uploading piece. err: <nil>
Dec 30 06:21:28.505337870 (590h52m28.376420915s) [noise=true , address=145.79.227.7:4294 , piece_id=L4ENKEYIDICT4SW7QTXB7IXSTXQ5XNK2HJQKCH7CVWD454BRND6Q, seg_pos={0 0}, upload=517989]: Done uploading piece. err: <nil>
Dec 30 06:21:28.505986014 (590h52m28.377069049s) [noise=true , address=145.79.227.10:4402 , piece_id=B3QDVFATNS3ELKAPMJMPL7VQ7REUZX75N35T3UFPTAJTIKSMY3VA, seg_pos={0 0}, upload=517989]: Done uploading piece. err: <nil>
Dec 30 06:21:28.507649481 (590h52m28.378732526s) [noise=true , address=145.79.227.6:4259 , piece_id=2H3JKA3KDOSALYHMV3TENUXY3SXD4EH5IXHETMRLO572FSCLMYMA, seg_pos={0 0}, upload=517989]: Done uploading piece. err: <nil>
Dec 30 06:21:28.513337461 (590h52m28.384420507s) [noise=true , address=145.79.227.47:4266 , piece_id=SF4O5OK7WLKI76F2AAP5QT4KH2GZ2L2CQP5HKHIS7SMU7S2BOEIQ, seg_pos={0 0}, upload=517989]: Done uploading piece. err: <nil>
Dec 30 06:21:28.527380693 (590h52m28.398463728s) [noise=true , address=145.79.227.20:4439 , piece_id=VANS6EWTKHBYGVKVIHYIVJKQBUVLSDOEV2VQ6TD4NEHTF5SJPQVA, seg_pos={0 0}, upload=517989]: Done uploading piece. err: <nil>
Dec 30 06:21:28.528615844 (590h52m28.399698899s) [noise=true , address=145.79.227.18:4006 , piece_id=JYGYYNDOQ6NCL6E3GECNBTUJ4V2LIDY5QGINEAVCZTAKO4LH66VA, seg_pos={0 0}, upload=517989]: Done uploading piece. err: <nil>
Dec 30 06:21:28.529479128 (590h52m28.400562173s) [noise=true , address=145.79.227.8:4329 , piece_id=ZXCFNCSW5H7ZPGBFC6LHX3RT5NZ6YHOTSDWV3GQVCYYQ7I57VADA, seg_pos={0 0}, upload=517989]: Done uploading piece. err: <nil>
Dec 30 06:21:28.530927807 (590h52m28.402010842s) [noise=true , address=145.79.227.50:4373 , piece_id=CHF3QXLYY3OPJHPHFHUEX4O4KSKE625O7BUJ7VZYSDTOHJRXRFPQ, seg_pos={0 0}, upload=517989]: Done uploading piece. err: <nil>
Dec 30 06:21:28.590999853 (590h52m28.462082888s) [noise=true , address=145.79.227.26:4654 , piece_id=MCTENZWHS5NYN3KQMANVEZKXN5KGTJDDT552IFI65IUU24DHTVLA, seg_pos={0 0}, upload=517989]: Done uploading piece. err: <nil>
Dec 30 06:21:28.706347903 (590h52m28.577430948s) [noise=true , address=77.90.31.15:4697 , piece_id=U6HUZIN25WB2CCPJ54QEKGUSFZ4KWE6FJ2EDLNT2VQMY2HJK3TCQ, seg_pos={0 0}, upload=517989]: Done uploading piece. err: <nil>
Dec 30 06:21:28.811628193 (590h52m28.682711228s) [noise=true , address=77.90.31.13:4627 , piece_id=TVFZBRPDXCKHVGMBKFOOMWONAZK5KJG7DLJ62FNQXM7T5P5AHB3Q, seg_pos={0 0}, upload=517989]: Done uploading piece. err: <nil>
Dec 30 06:21:28.811652279 (590h52m28.682735314s) [seg_pos={0 0}, upload=517989]: Segment reached optimal threshold of 13 pieces.
Dec 30 06:21:28.811830728 (590h52m28.682913764s) [noise=true , address=45.91.116.8:4047 , piece_id=GSG2ZHOR56FAWWZAH5EUHUNQ3SS6IHJ6ZZMVZHM2HQFN3XYTYYCA, seg_pos={0 0}, upload=517989]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF
Dec 30 06:21:28.811837461 (590h52m28.682920496s) [noise=true , address=45.91.116.8:4047 , piece_id=GSG2ZHOR56FAWWZAH5EUHUNQ3SS6IHJ6ZZMVZHM2HQFN3XYTYYCA, seg_pos={0 0}, upload=517989]: Long tail cancellation due to optimal threshold
Dec 30 06:21:28.812030229 (590h52m28.683113274s) [noise=true , address=45.91.116.7:4009 , piece_id=IIOD3ELLRYAL7JXXADWX6TU4KPDA55ZH543CBCSVZYOLMFS2LPVA, seg_pos={0 0}, upload=517989]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF
Dec 30 06:21:28.812047822 (590h52m28.683130857s) [noise=true , address=45.91.116.7:4009 , piece_id=IIOD3ELLRYAL7JXXADWX6TU4KPDA55ZH543CBCSVZYOLMFS2LPVA, seg_pos={0 0}, upload=517989]: Long tail cancellation due to optimal threshold
Dec 30 06:21:28.812061889 (590h52m28.683144934s) [noise=true , address=45.91.116.5:4300 , piece_id=OKCDW5TU2YBOWJO6FGB4K6Q5DWTLPP2ZP257BORAHO5CZK27ODEQ, seg_pos={0 0}, upload=517989]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:47874->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:47874->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47874->45.91.116.5:4300: use of closed network connection
Dec 30 06:21:28.812069313 (590h52m28.683152348s) [noise=true , address=45.91.116.5:4300 , piece_id=OKCDW5TU2YBOWJO6FGB4K6Q5DWTLPP2ZP257BORAHO5CZK27ODEQ, seg_pos={0 0}, upload=517989]: Long tail cancellation due to optimal threshold
Dec 30 06:21:28.812087727 (590h52m28.683170773s) [seg_pos={0 0}, upload=517989]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 06:21:28.812222985 (590h52m28.68330603s) [upload=517989]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:21:28.812232614 (590h52m28.683315669s) [upload=517989]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:21:29.101994995 (590h52m28.97307804s) [upload=517989]: Done uploading segments...
Dec 30 06:21:29.102016356 (590h52m28.973099401s) [upload=517989]: Upload finished. err: <nil>
Duration: 5.984866254s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....................................................X.......] duration:
7.255630041s trace:
https://jaeger.dev.storj.io/trace/deb96436003dee
FAIL (2025-12-30 06:22:13 UTC)
orbiter warning: read tcp 188.166.225.170:52458->45.38.24.2:443: read: connection reset by peer
Duration: 7.255659998s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [....................................X...X......X............] duration:
25.048945709s trace:
https://jaeger.dev.storj.io/trace/46b48df111d99581
FAIL (2025-12-30 06:22:10 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
402256s) [noise=true , address=77.90.31.17:4794 , piece_id=5FI7ZF2GFCHPDKEB7F3U5NMXZFGAJEDUFVZA5H4U5HJJI2OTB5EQ, seg_pos={0 0}, upload=517864]: Uploading piece...
Dec 30 06:13:02.896086983 (590h44m2.767170058s) [noise=true , address=77.90.31.19:4857 , piece_id=WGGMF7ZW4HJROMZ3HT7PHY3LWSFFGQSZNDXZYFMIY4LIA7736FSA, seg_pos={0 0}, upload=517864]: Uploading piece...
Dec 30 06:13:02.896896775 (590h44m2.76797982s) [noise=true , address=145.79.227.34:4919 , piece_id=K5HTPRIPPAJBRFLGJMTXMUBQYP2VUKSSFBIDZJ42YXKGZ6NRHP7Q, seg_pos={0 0}, upload=517864]: Uploading piece...
Dec 30 06:13:02.897704834 (590h44m2.768787869s) [noise=true , address=64.49.13.169:4709 , piece_id=4QGR676MN3UCO7VOCOLCSMY2EEVDN4E2BQEZ6T73BBBUFEY7F7CQ, seg_pos={0 0}, upload=517864]: Uploading piece...
Dec 30 06:13:02.898734705 (590h44m2.76981775s) [noise=true , address=77.90.31.11:4580 , piece_id=X5XOX26FSYNEM6E5Y2FHBVSOBDVU5XJ6Z7XFRSPQSHIOPIM4ILLQ, seg_pos={0 0}, upload=517864]: Uploading piece...
Dec 30 06:13:02.900012478 (590h44m2.771095523s) [noise=true , address=193.149.164.143:4699 , piece_id=MVZB6VSTMP3J44LDQDK7QXIA27AB3XLPIGYNHLZDXJODYOSTN3EQ, seg_pos={0 0}, upload=517864]: Uploading piece...
Dec 30 06:13:02.902485407 (590h44m2.773568462s) [noise=true , address=64.49.13.169:4716 , piece_id=3HZZKO6XDXO2JQCUOIXIWUKWRQ3WESRYW3TPRQWTFTM6UYUE4VOQ, seg_pos={0 0}, upload=517864]: Uploading piece...
Dec 30 06:13:02.902823431 (590h44m2.773906476s) [noise=true , address=77.90.31.14:4688 , piece_id=XJU7EHH2573FOVLOSLGAA46JEZG2AIKLVKZOXRBYYM7X56E22RGA, seg_pos={0 0}, upload=517864]: Uploading piece...
Dec 30 06:13:02.903164710 (590h44m2.774247755s) [noise=true , address=45.91.116.11:4139 , piece_id=CKEVNQMM5MAMUE4XHWYN6SD4MWZB5CMMYAB7JDZ3C43NNMDSJVCA, seg_pos={0 0}, upload=517864]: Uploading piece...
Dec 30 06:13:02.903544323 (590h44m2.774627358s) [noise=true , address=145.79.227.30:4771 , piece_id=U3PVHRB7KCA76GXAO3SJJOED3NUEJ3B3TXXQE6SX4DJEM6M5O4RA, seg_pos={0 0}, upload=517864]: Uploading piece...
Dec 30 06:13:02.905176611 (590h44m2.776259656s) [noise=true , address=145.79.227.11:4735 , piece_id=ENOSVX5NS3JUUCILOYYKHDZHOUSJZD6THCW365EYZLOY3BSE6D5A, seg_pos={0 0}, upload=517864]: Uploading piece...
Dec 30 06:13:02.906736873 (590h44m2.777819908s) [noise=true , address=77.90.31.15:4717 , piece_id=4X6UXT3HTTALGUBA4VAIMQS6P6P5JLY7FVDAFWL2PK2DUVZDBPZA, seg_pos={0 0}, upload=517864]: Uploading piece...
Dec 30 06:13:02.907198291 (590h44m2.778281336s) [noise=true , address=77.90.31.18:4833 , piece_id=5OHYRK74NRJAS6ENJIEEP3NXETW2GIOHESUCHHLIUCOO5E3TENPQ, seg_pos={0 0}, upload=517864]: Uploading piece...
Dec 30 06:13:02.908486875 (590h44m2.77956991s) [noise=true , address=145.79.227.33:4898 , piece_id=4J7VGL3KNWH3GZ4ON4KXZLLCZLBUGHROKIIB7WGY63RHH5IPF2IA, seg_pos={0 0}, upload=517864]: Uploading piece...
Dec 30 06:13:02.908969063 (590h44m2.780052108s) [noise=true , address=145.79.227.13:4832 , piece_id=JVI5TVAKEYPCIQJ4465JGHH4DO6UR6VJTAAIECE7EWVECNBNCIPA, seg_pos={0 0}, upload=517864]: Uploading piece...
Dec 30 06:13:04.360994985 (590h44m4.23207803s) [upload=517864]: Next returned nil segment
Dec 30 06:13:04.361008010 (590h44m4.232091045s) [upload=517864]: Waiting for error group managing segments...
Dec 30 06:13:04.781371641 (590h44m4.652454686s) [noise=true , address=145.79.227.34:4919 , piece_id=K5HTPRIPPAJBRFLGJMTXMUBQYP2VUKSSFBIDZJ42YXKGZ6NRHP7Q, seg_pos={0 0}, upload=517864]: Done uploading piece. err: <nil>
Dec 30 06:13:04.784208242 (590h44m4.655291288s) [noise=true , address=145.79.227.11:4735 , piece_id=ENOSVX5NS3JUUCILOYYKHDZHOUSJZD6THCW365EYZLOY3BSE6D5A, seg_pos={0 0}, upload=517864]: Done uploading piece. err: <nil>
Dec 30 06:13:04.836043685 (590h44m4.70712673s) [noise=true , address=145.79.227.35:4486 , piece_id=KW3PACCWHZXZPQ5TKPSRSYFSHBXN5XMVH3XXOVEZUR3SF3JFOUVA, seg_pos={0 0}, upload=517864]: Done uploading piece. err: <nil>
Dec 30 06:13:04.866039612 (590h44m4.737122657s) [noise=true , address=145.79.227.13:4832 , piece_id=JVI5TVAKEYPCIQJ4465JGHH4DO6UR6VJTAAIECE7EWVECNBNCIPA, seg_pos={0 0}, upload=517864]: Done uploading piece. err: <nil>
Dec 30 06:13:05.063568059 (590h44m4.934651124s) [noise=true , address=77.90.31.11:4580 , piece_id=X5XOX26FSYNEM6E5Y2FHBVSOBDVU5XJ6Z7XFRSPQSHIOPIM4ILLQ, seg_pos={0 0}, upload=517864]: Done uploading piece. err: <nil>
Dec 30 06:13:05.071636672 (590h44m4.942719727s) [noise=true , address=77.90.31.19:4857 , piece_id=WGGMF7ZW4HJROMZ3HT7PHY3LWSFFGQSZNDXZYFMIY4LIA7736FSA, seg_pos={0 0}, upload=517864]: Done uploading piece. err: <nil>
Dec 30 06:13:05.088316663 (590h44m4.959399708s) [noise=true , address=77.90.31.17:4794 , piece_id=5FI7ZF2GFCHPDKEB7F3U5NMXZFGAJEDUFVZA5H4U5HJJI2OTB5EQ, seg_pos={0 0}, upload=517864]: Done uploading piece. err: <nil>
Dec 30 06:13:05.170834401 (590h44m5.041917446s) [noise=true , address=77.90.31.15:4717 , piece_id=4X6UXT3HTTALGUBA4VAIMQS6P6P5JLY7FVDAFWL2PK2DUVZDBPZA, seg_pos={0 0}, upload=517864]: Done uploading piece. err: <nil>
Dec 30 06:13:05.272783788 (590h44m5.143866834s) [noise=true , address=77.90.31.14:4688 , piece_id=XJU7EHH2573FOVLOSLGAA46JEZG2AIKLVKZOXRBYYM7X56E22RGA, seg_pos={0 0}, upload=517864]: Done uploading piece. err: <nil>
Dec 30 06:13:05.667537083 (590h44m5.538620128s) [noise=true , address=145.79.227.43:4189 , piece_id=FZK7EWS2AKZZZL4B45TAR3SVPJ2UYFJGR35HIIPU3FUEWHRSX36A, seg_pos={0 0}, upload=517864]: Done uploading piece. err: <nil>
Dec 30 06:13:05.691236510 (590h44m5.562319555s) [noise=true , address=193.149.164.143:4699 , piece_id=MVZB6VSTMP3J44LDQDK7QXIA27AB3XLPIGYNHLZDXJODYOSTN3EQ, seg_pos={0 0}, upload=517864]: Done uploading piece. err: <nil>
Dec 30 06:13:05.712297864 (590h44m5.583380899s) [noise=true , address=145.79.227.30:4771 , piece_id=U3PVHRB7KCA76GXAO3SJJOED3NUEJ3B3TXXQE6SX4DJEM6M5O4RA, seg_pos={0 0}, upload=517864]: Done uploading piece. err: <nil>
Dec 30 06:13:05.720879355 (590h44m5.591962401s) [noise=true , address=145.79.227.40:4082 , piece_id=4BS3GCGUFDSZLFXPPXAPT2524TJRLZT6GMH6Y76HLIFQDHXNPKBA, seg_pos={0 0}, upload=517864]: Done uploading piece. err: <nil>
Dec 30 06:13:05.954725096 (590h44m5.825808141s) [noise=true , address=145.79.227.33:4898 , piece_id=4J7VGL3KNWH3GZ4ON4KXZLLCZLBUGHROKIIB7WGY63RHH5IPF2IA, seg_pos={0 0}, upload=517864]: Done uploading piece. err: <nil>
Dec 30 06:13:05.954752869 (590h44m5.825835914s) [seg_pos={0 0}, upload=517864]: Segment reached optimal threshold of 14 pieces.
Dec 30 06:13:05.955016099 (590h44m5.826099144s) [noise=true , address=64.49.13.169:4709 , piece_id=4QGR676MN3UCO7VOCOLCSMY2EEVDN4E2BQEZ6T73BBBUFEY7F7CQ, seg_pos={0 0}, upload=517864]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:49652->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:49652->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49652->64.49.13.169:4709: use of closed network connection
Dec 30 06:13:05.955027190 (590h44m5.826110225s) [noise=true , address=64.49.13.169:4709 , piece_id=4QGR676MN3UCO7VOCOLCSMY2EEVDN4E2BQEZ6T73BBBUFEY7F7CQ, seg_pos={0 0}, upload=517864]: Long tail cancellation due to optimal threshold
Dec 30 06:13:05.955054612 (590h44m5.826137658s) [noise=true , address=45.91.116.11:4139 , piece_id=CKEVNQMM5MAMUE4XHWYN6SD4MWZB5CMMYAB7JDZ3C43NNMDSJVCA, seg_pos={0 0}, upload=517864]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Tz3HEHACmHU9Gp2W2vHVuF5PXNmdsa5N8nYhwt52HCTjggqRD): protocol: expected piece hash; context canceled; EOF
Dec 30 06:13:05.955062788 (590h44m5.826145833s) [noise=true , address=45.91.116.11:4139 , piece_id=CKEVNQMM5MAMUE4XHWYN6SD4MWZB5CMMYAB7JDZ3C43NNMDSJVCA, seg_pos={0 0}, upload=517864]: Long tail cancellation due to optimal threshold
Dec 30 06:13:05.955166545 (590h44m5.82624959s) [noise=true , address=77.90.31.18:4833 , piece_id=5OHYRK74NRJAS6ENJIEEP3NXETW2GIOHESUCHHLIUCOO5E3TENPQ, seg_pos={0 0}, upload=517864]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FbVbo8kGy1aaFyCk2noiXNUJHtosFJHFykBMRecGd5Kj5gjEp): protocol: expected piece hash; context canceled; EOF
Dec 30 06:13:05.955172297 (590h44m5.826255342s) [noise=true , address=77.90.31.18:4833 , piece_id=5OHYRK74NRJAS6ENJIEEP3NXETW2GIOHESUCHHLIUCOO5E3TENPQ, seg_pos={0 0}, upload=517864]: Long tail cancellation due to optimal threshold
Dec 30 06:13:05.955273479 (590h44m5.826356524s) [noise=true , address=64.49.13.169:4716 , piece_id=3HZZKO6XDXO2JQCUOIXIWUKWRQ3WESRYW3TPRQWTFTM6UYUE4VOQ, seg_pos={0 0}, upload=517864]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:49808->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:49808->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49808->64.49.13.169:4716: use of closed network connection
Dec 30 06:13:05.955280313 (590h44m5.826363348s) [noise=true , address=64.49.13.169:4716 , piece_id=3HZZKO6XDXO2JQCUOIXIWUKWRQ3WESRYW3TPRQWTFTM6UYUE4VOQ, seg_pos={0 0}, upload=517864]: Long tail cancellation due to optimal threshold
Dec 30 06:13:05.955310550 (590h44m5.826393596s) [seg_pos={0 0}, upload=517864]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 06:13:05.955381786 (590h44m5.826464821s) [upload=517864]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:13:05.955386705 (590h44m5.82646974s) [upload=517864]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:13:06.243506482 (590h44m6.114589527s) [upload=517864]: Done uploading segments...
Dec 30 06:13:06.243537802 (590h44m6.114620837s) [upload=517864]: Upload finished. err: <nil>
Duration: 25.048975826s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [....................................X...........X...........] duration:
3.455560552s trace:
https://jaeger.dev.storj.io/trace/19740e6f10941c1
FAIL (2025-12-30 06:21:56 UTC)
orbiter warning: context deadline exceeded
Duration: 3.455688465s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
18 out of 60 is failed (threshold=3) [X..X..X....X.XX.X..X..X..X..XX.......X..X......X....X...X..X] duration:
30.00258194s trace:
https://jaeger.dev.storj.io/trace/65e658867ff66eaf
FAIL (2025-12-30 06:21:55 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1VawvPHa7gM65pn3kxPmaMjmXqMNrutEKXTmbDxdeNP9WCgLSB, Piece ID: TVFZBRPDXCKHVGMBKFOOMWONAZK5KJG7DLJ62FNQXM7T5P5AHB3Q) context deadline exceeded; piecestore: (Node ID: 12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, Piece ID: U6HUZIN25WB2CCPJ54QEKGUSFZ4KWE6FJ2EDLNT2VQMY2HJK3TCQ) context deadline exceeded
======LOGS======
30 06:21:26.267761284 (590h52m26.138844319s) [upload=517989]: Starting upload
Dec 30 06:21:26.267835385 (590h52m26.13891844s) [upload=517989]: Done starting upload
Dec 30 06:21:26.267927901 (590h52m26.139010936s) [upload=517989]: Uploading segments...
Dec 30 06:21:26.268098546 (590h52m26.139181591s) [upload=517989]: Got next segment. Inline: false
Dec 30 06:21:26.268143722 (590h52m26.139226757s) [upload=517989]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 06:21:26.268152128 (590h52m26.139235163s) [upload=517989]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 06:21:26.515138137 (590h52m26.386221182s) [seg_pos={0 0}, upload=517989]: Begin upload segment...
Dec 30 06:21:26.515190235 (590h52m26.386273281s) [seg_pos={0 0}, upload=517989]: Done begin upload segment.
Dec 30 06:21:26.515226214 (590h52m26.386309259s) [noise=true , address=45.91.116.7:4009 , piece_id=IIOD3ELLRYAL7JXXADWX6TU4KPDA55ZH543CBCSVZYOLMFS2LPVA, seg_pos={0 0}, upload=517989]: Uploading piece...
Dec 30 06:21:26.515415906 (590h52m26.386498951s) [noise=true , address=145.79.227.15:4876 , piece_id=VP27XUZSPN3CWTWSK7Z67ESPR4K6QG2IZ3F44NRRFVVIE4SYWM2A, seg_pos={0 0}, upload=517989]: Uploading piece...
Dec 30 06:21:26.515955713 (590h52m26.387038748s) [noise=true , address=145.79.227.10:4402 , piece_id=B3QDVFATNS3ELKAPMJMPL7VQ7REUZX75N35T3UFPTAJTIKSMY3VA, seg_pos={0 0}, upload=517989]: Uploading piece...
Dec 30 06:21:26.516497335 (590h52m26.38758038s) [noise=true , address=145.79.227.6:4259 , piece_id=2H3JKA3KDOSALYHMV3TENUXY3SXD4EH5IXHETMRLO572FSCLMYMA, seg_pos={0 0}, upload=517989]: Uploading piece...
Dec 30 06:21:26.516929918 (590h52m26.388012963s) [noise=true , address=145.79.227.34:4907 , piece_id=L76LLBAPKNPYEHZWXRTOECRHRUU6VMEDCEWCIYVULFTUWGJCG5QA, seg_pos={0 0}, upload=517989]: Uploading piece...
Dec 30 06:21:26.517317215 (590h52m26.388400251s) [noise=true , address=145.79.227.26:4654 , piece_id=MCTENZWHS5NYN3KQMANVEZKXN5KGTJDDT552IFI65IUU24DHTVLA, seg_pos={0 0}, upload=517989]: Uploading piece...
Dec 30 06:21:26.517684736 (590h52m26.388767771s) [noise=true , address=145.79.227.47:4266 , piece_id=SF4O5OK7WLKI76F2AAP5QT4KH2GZ2L2CQP5HKHIS7SMU7S2BOEIQ, seg_pos={0 0}, upload=517989]: Uploading piece...
Dec 30 06:21:26.517703962 (590h52m26.388787007s) [noise=true , address=77.90.31.15:4697 , piece_id=U6HUZIN25WB2CCPJ54QEKGUSFZ4KWE6FJ2EDLNT2VQMY2HJK3TCQ, seg_pos={0 0}, upload=517989]: Uploading piece...
Dec 30 06:21:26.518244662 (590h52m26.389327707s) [noise=true , address=145.79.227.7:4294 , piece_id=L4ENKEYIDICT4SW7QTXB7IXSTXQ5XNK2HJQKCH7CVWD454BRND6Q, seg_pos={0 0}, upload=517989]: Uploading piece...
Dec 30 06:21:26.518310287 (590h52m26.389393322s) [noise=true , address=145.79.227.50:4373 , piece_id=CHF3QXLYY3OPJHPHFHUEX4O4KSKE625O7BUJ7VZYSDTOHJRXRFPQ, seg_pos={0 0}, upload=517989]: Uploading piece...
Dec 30 06:21:26.518649933 (590h52m26.389732978s) [noise=true , address=145.79.227.18:4006 , piece_id=JYGYYNDOQ6NCL6E3GECNBTUJ4V2LIDY5QGINEAVCZTAKO4LH66VA, seg_pos={0 0}, upload=517989]: Uploading piece...
Dec 30 06:21:26.518857398 (590h52m26.389940443s) [noise=true , address=145.79.227.8:4329 , piece_id=ZXCFNCSW5H7ZPGBFC6LHX3RT5NZ6YHOTSDWV3GQVCYYQ7I57VADA, seg_pos={0 0}, upload=517989]: Uploading piece...
Dec 30 06:21:26.519034176 (590h52m26.390117221s) [noise=true , address=145.79.227.20:4439 , piece_id=VANS6EWTKHBYGVKVIHYIVJKQBUVLSDOEV2VQ6TD4NEHTF5SJPQVA, seg_pos={0 0}, upload=517989]: Uploading piece...
Dec 30 06:21:26.519374443 (590h52m26.390457478s) [noise=true , address=45.91.116.5:4300 , piece_id=OKCDW5TU2YBOWJO6FGB4K6Q5DWTLPP2ZP257BORAHO5CZK27ODEQ, seg_pos={0 0}, upload=517989]: Uploading piece...
Dec 30 06:21:26.519907659 (590h52m26.390990704s) [noise=true , address=77.90.31.13:4627 , piece_id=TVFZBRPDXCKHVGMBKFOOMWONAZK5KJG7DLJ62FNQXM7T5P5AHB3Q, seg_pos={0 0}, upload=517989]: Uploading piece...
Dec 30 06:21:26.520439221 (590h52m26.391522256s) [noise=true , address=45.91.116.8:4047 , piece_id=GSG2ZHOR56FAWWZAH5EUHUNQ3SS6IHJ6ZZMVZHM2HQFN3XYTYYCA, seg_pos={0 0}, upload=517989]: Uploading piece...
Dec 30 06:21:28.214165843 (590h52m28.085248878s) [upload=517989]: Next returned nil segment
Dec 30 06:21:28.214179650 (590h52m28.085262685s) [upload=517989]: Waiting for error group managing segments...
Dec 30 06:21:28.498112992 (590h52m28.369196037s) [noise=true , address=145.79.227.34:4907 , piece_id=L76LLBAPKNPYEHZWXRTOECRHRUU6VMEDCEWCIYVULFTUWGJCG5QA, seg_pos={0 0}, upload=517989]: Done uploading piece. err: <nil>
Dec 30 06:21:28.505291993 (590h52m28.376375038s) [noise=true , address=145.79.227.15:4876 , piece_id=VP27XUZSPN3CWTWSK7Z67ESPR4K6QG2IZ3F44NRRFVVIE4SYWM2A, seg_pos={0 0}, upload=517989]: Done uploading piece. err: <nil>
Dec 30 06:21:28.505337870 (590h52m28.376420915s) [noise=true , address=145.79.227.7:4294 , piece_id=L4ENKEYIDICT4SW7QTXB7IXSTXQ5XNK2HJQKCH7CVWD454BRND6Q, seg_pos={0 0}, upload=517989]: Done uploading piece. err: <nil>
Dec 30 06:21:28.505986014 (590h52m28.377069049s) [noise=true , address=145.79.227.10:4402 , piece_id=B3QDVFATNS3ELKAPMJMPL7VQ7REUZX75N35T3UFPTAJTIKSMY3VA, seg_pos={0 0}, upload=517989]: Done uploading piece. err: <nil>
Dec 30 06:21:28.507649481 (590h52m28.378732526s) [noise=true , address=145.79.227.6:4259 , piece_id=2H3JKA3KDOSALYHMV3TENUXY3SXD4EH5IXHETMRLO572FSCLMYMA, seg_pos={0 0}, upload=517989]: Done uploading piece. err: <nil>
Dec 30 06:21:28.513337461 (590h52m28.384420507s) [noise=true , address=145.79.227.47:4266 , piece_id=SF4O5OK7WLKI76F2AAP5QT4KH2GZ2L2CQP5HKHIS7SMU7S2BOEIQ, seg_pos={0 0}, upload=517989]: Done uploading piece. err: <nil>
Dec 30 06:21:28.527380693 (590h52m28.398463728s) [noise=true , address=145.79.227.20:4439 , piece_id=VANS6EWTKHBYGVKVIHYIVJKQBUVLSDOEV2VQ6TD4NEHTF5SJPQVA, seg_pos={0 0}, upload=517989]: Done uploading piece. err: <nil>
Dec 30 06:21:28.528615844 (590h52m28.399698899s) [noise=true , address=145.79.227.18:4006 , piece_id=JYGYYNDOQ6NCL6E3GECNBTUJ4V2LIDY5QGINEAVCZTAKO4LH66VA, seg_pos={0 0}, upload=517989]: Done uploading piece. err: <nil>
Dec 30 06:21:28.529479128 (590h52m28.400562173s) [noise=true , address=145.79.227.8:4329 , piece_id=ZXCFNCSW5H7ZPGBFC6LHX3RT5NZ6YHOTSDWV3GQVCYYQ7I57VADA, seg_pos={0 0}, upload=517989]: Done uploading piece. err: <nil>
Dec 30 06:21:28.530927807 (590h52m28.402010842s) [noise=true , address=145.79.227.50:4373 , piece_id=CHF3QXLYY3OPJHPHFHUEX4O4KSKE625O7BUJ7VZYSDTOHJRXRFPQ, seg_pos={0 0}, upload=517989]: Done uploading piece. err: <nil>
Dec 30 06:21:28.590999853 (590h52m28.462082888s) [noise=true , address=145.79.227.26:4654 , piece_id=MCTENZWHS5NYN3KQMANVEZKXN5KGTJDDT552IFI65IUU24DHTVLA, seg_pos={0 0}, upload=517989]: Done uploading piece. err: <nil>
Dec 30 06:21:28.706347903 (590h52m28.577430948s) [noise=true , address=77.90.31.15:4697 , piece_id=U6HUZIN25WB2CCPJ54QEKGUSFZ4KWE6FJ2EDLNT2VQMY2HJK3TCQ, seg_pos={0 0}, upload=517989]: Done uploading piece. err: <nil>
Dec 30 06:21:28.811628193 (590h52m28.682711228s) [noise=true , address=77.90.31.13:4627 , piece_id=TVFZBRPDXCKHVGMBKFOOMWONAZK5KJG7DLJ62FNQXM7T5P5AHB3Q, seg_pos={0 0}, upload=517989]: Done uploading piece. err: <nil>
Dec 30 06:21:28.811652279 (590h52m28.682735314s) [seg_pos={0 0}, upload=517989]: Segment reached optimal threshold of 13 pieces.
Dec 30 06:21:28.811830728 (590h52m28.682913764s) [noise=true , address=45.91.116.8:4047 , piece_id=GSG2ZHOR56FAWWZAH5EUHUNQ3SS6IHJ6ZZMVZHM2HQFN3XYTYYCA, seg_pos={0 0}, upload=517989]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN): protocol: expected piece hash; context canceled; EOF
Dec 30 06:21:28.811837461 (590h52m28.682920496s) [noise=true , address=45.91.116.8:4047 , piece_id=GSG2ZHOR56FAWWZAH5EUHUNQ3SS6IHJ6ZZMVZHM2HQFN3XYTYYCA, seg_pos={0 0}, upload=517989]: Long tail cancellation due to optimal threshold
Dec 30 06:21:28.812030229 (590h52m28.683113274s) [noise=true , address=45.91.116.7:4009 , piece_id=IIOD3ELLRYAL7JXXADWX6TU4KPDA55ZH543CBCSVZYOLMFS2LPVA, seg_pos={0 0}, upload=517989]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KaUBuZoi8DjagfvnxBSLBL1Mt69RdtNKn96oT7wFC8n2ppFDE): protocol: expected piece hash; context canceled; EOF
Dec 30 06:21:28.812047822 (590h52m28.683130857s) [noise=true , address=45.91.116.7:4009 , piece_id=IIOD3ELLRYAL7JXXADWX6TU4KPDA55ZH543CBCSVZYOLMFS2LPVA, seg_pos={0 0}, upload=517989]: Long tail cancellation due to optimal threshold
Dec 30 06:21:28.812061889 (590h52m28.683144934s) [noise=true , address=45.91.116.5:4300 , piece_id=OKCDW5TU2YBOWJO6FGB4K6Q5DWTLPP2ZP257BORAHO5CZK27ODEQ, seg_pos={0 0}, upload=517989]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: write tcp 188.166.225.170:47874->45.91.116.5:4300: use of closed network connection; write tcp 188.166.225.170:47874->45.91.116.5:4300: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:47874->45.91.116.5:4300: use of closed network connection
Dec 30 06:21:28.812069313 (590h52m28.683152348s) [noise=true , address=45.91.116.5:4300 , piece_id=OKCDW5TU2YBOWJO6FGB4K6Q5DWTLPP2ZP257BORAHO5CZK27ODEQ, seg_pos={0 0}, upload=517989]: Long tail cancellation due to optimal threshold
Dec 30 06:21:28.812087727 (590h52m28.683170773s) [seg_pos={0 0}, upload=517989]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 06:21:28.812222985 (590h52m28.68330603s) [upload=517989]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:21:28.812232614 (590h52m28.683315669s) [upload=517989]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:21:29.101994995 (590h52m28.97307804s) [upload=517989]: Done uploading segments...
Dec 30 06:21:29.102016356 (590h52m28.973099401s) [upload=517989]: Upload finished. err: <nil>
Duration: 30.002602841s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [.....................................X...X......X...........] duration:
42.433022566s trace:
https://jaeger.dev.storj.io/trace/46b48df111d99581
FAIL (2025-12-30 06:21:36 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
402256s) [noise=true , address=77.90.31.17:4794 , piece_id=5FI7ZF2GFCHPDKEB7F3U5NMXZFGAJEDUFVZA5H4U5HJJI2OTB5EQ, seg_pos={0 0}, upload=517864]: Uploading piece...
Dec 30 06:13:02.896086983 (590h44m2.767170058s) [noise=true , address=77.90.31.19:4857 , piece_id=WGGMF7ZW4HJROMZ3HT7PHY3LWSFFGQSZNDXZYFMIY4LIA7736FSA, seg_pos={0 0}, upload=517864]: Uploading piece...
Dec 30 06:13:02.896896775 (590h44m2.76797982s) [noise=true , address=145.79.227.34:4919 , piece_id=K5HTPRIPPAJBRFLGJMTXMUBQYP2VUKSSFBIDZJ42YXKGZ6NRHP7Q, seg_pos={0 0}, upload=517864]: Uploading piece...
Dec 30 06:13:02.897704834 (590h44m2.768787869s) [noise=true , address=64.49.13.169:4709 , piece_id=4QGR676MN3UCO7VOCOLCSMY2EEVDN4E2BQEZ6T73BBBUFEY7F7CQ, seg_pos={0 0}, upload=517864]: Uploading piece...
Dec 30 06:13:02.898734705 (590h44m2.76981775s) [noise=true , address=77.90.31.11:4580 , piece_id=X5XOX26FSYNEM6E5Y2FHBVSOBDVU5XJ6Z7XFRSPQSHIOPIM4ILLQ, seg_pos={0 0}, upload=517864]: Uploading piece...
Dec 30 06:13:02.900012478 (590h44m2.771095523s) [noise=true , address=193.149.164.143:4699 , piece_id=MVZB6VSTMP3J44LDQDK7QXIA27AB3XLPIGYNHLZDXJODYOSTN3EQ, seg_pos={0 0}, upload=517864]: Uploading piece...
Dec 30 06:13:02.902485407 (590h44m2.773568462s) [noise=true , address=64.49.13.169:4716 , piece_id=3HZZKO6XDXO2JQCUOIXIWUKWRQ3WESRYW3TPRQWTFTM6UYUE4VOQ, seg_pos={0 0}, upload=517864]: Uploading piece...
Dec 30 06:13:02.902823431 (590h44m2.773906476s) [noise=true , address=77.90.31.14:4688 , piece_id=XJU7EHH2573FOVLOSLGAA46JEZG2AIKLVKZOXRBYYM7X56E22RGA, seg_pos={0 0}, upload=517864]: Uploading piece...
Dec 30 06:13:02.903164710 (590h44m2.774247755s) [noise=true , address=45.91.116.11:4139 , piece_id=CKEVNQMM5MAMUE4XHWYN6SD4MWZB5CMMYAB7JDZ3C43NNMDSJVCA, seg_pos={0 0}, upload=517864]: Uploading piece...
Dec 30 06:13:02.903544323 (590h44m2.774627358s) [noise=true , address=145.79.227.30:4771 , piece_id=U3PVHRB7KCA76GXAO3SJJOED3NUEJ3B3TXXQE6SX4DJEM6M5O4RA, seg_pos={0 0}, upload=517864]: Uploading piece...
Dec 30 06:13:02.905176611 (590h44m2.776259656s) [noise=true , address=145.79.227.11:4735 , piece_id=ENOSVX5NS3JUUCILOYYKHDZHOUSJZD6THCW365EYZLOY3BSE6D5A, seg_pos={0 0}, upload=517864]: Uploading piece...
Dec 30 06:13:02.906736873 (590h44m2.777819908s) [noise=true , address=77.90.31.15:4717 , piece_id=4X6UXT3HTTALGUBA4VAIMQS6P6P5JLY7FVDAFWL2PK2DUVZDBPZA, seg_pos={0 0}, upload=517864]: Uploading piece...
Dec 30 06:13:02.907198291 (590h44m2.778281336s) [noise=true , address=77.90.31.18:4833 , piece_id=5OHYRK74NRJAS6ENJIEEP3NXETW2GIOHESUCHHLIUCOO5E3TENPQ, seg_pos={0 0}, upload=517864]: Uploading piece...
Dec 30 06:13:02.908486875 (590h44m2.77956991s) [noise=true , address=145.79.227.33:4898 , piece_id=4J7VGL3KNWH3GZ4ON4KXZLLCZLBUGHROKIIB7WGY63RHH5IPF2IA, seg_pos={0 0}, upload=517864]: Uploading piece...
Dec 30 06:13:02.908969063 (590h44m2.780052108s) [noise=true , address=145.79.227.13:4832 , piece_id=JVI5TVAKEYPCIQJ4465JGHH4DO6UR6VJTAAIECE7EWVECNBNCIPA, seg_pos={0 0}, upload=517864]: Uploading piece...
Dec 30 06:13:04.360994985 (590h44m4.23207803s) [upload=517864]: Next returned nil segment
Dec 30 06:13:04.361008010 (590h44m4.232091045s) [upload=517864]: Waiting for error group managing segments...
Dec 30 06:13:04.781371641 (590h44m4.652454686s) [noise=true , address=145.79.227.34:4919 , piece_id=K5HTPRIPPAJBRFLGJMTXMUBQYP2VUKSSFBIDZJ42YXKGZ6NRHP7Q, seg_pos={0 0}, upload=517864]: Done uploading piece. err: <nil>
Dec 30 06:13:04.784208242 (590h44m4.655291288s) [noise=true , address=145.79.227.11:4735 , piece_id=ENOSVX5NS3JUUCILOYYKHDZHOUSJZD6THCW365EYZLOY3BSE6D5A, seg_pos={0 0}, upload=517864]: Done uploading piece. err: <nil>
Dec 30 06:13:04.836043685 (590h44m4.70712673s) [noise=true , address=145.79.227.35:4486 , piece_id=KW3PACCWHZXZPQ5TKPSRSYFSHBXN5XMVH3XXOVEZUR3SF3JFOUVA, seg_pos={0 0}, upload=517864]: Done uploading piece. err: <nil>
Dec 30 06:13:04.866039612 (590h44m4.737122657s) [noise=true , address=145.79.227.13:4832 , piece_id=JVI5TVAKEYPCIQJ4465JGHH4DO6UR6VJTAAIECE7EWVECNBNCIPA, seg_pos={0 0}, upload=517864]: Done uploading piece. err: <nil>
Dec 30 06:13:05.063568059 (590h44m4.934651124s) [noise=true , address=77.90.31.11:4580 , piece_id=X5XOX26FSYNEM6E5Y2FHBVSOBDVU5XJ6Z7XFRSPQSHIOPIM4ILLQ, seg_pos={0 0}, upload=517864]: Done uploading piece. err: <nil>
Dec 30 06:13:05.071636672 (590h44m4.942719727s) [noise=true , address=77.90.31.19:4857 , piece_id=WGGMF7ZW4HJROMZ3HT7PHY3LWSFFGQSZNDXZYFMIY4LIA7736FSA, seg_pos={0 0}, upload=517864]: Done uploading piece. err: <nil>
Dec 30 06:13:05.088316663 (590h44m4.959399708s) [noise=true , address=77.90.31.17:4794 , piece_id=5FI7ZF2GFCHPDKEB7F3U5NMXZFGAJEDUFVZA5H4U5HJJI2OTB5EQ, seg_pos={0 0}, upload=517864]: Done uploading piece. err: <nil>
Dec 30 06:13:05.170834401 (590h44m5.041917446s) [noise=true , address=77.90.31.15:4717 , piece_id=4X6UXT3HTTALGUBA4VAIMQS6P6P5JLY7FVDAFWL2PK2DUVZDBPZA, seg_pos={0 0}, upload=517864]: Done uploading piece. err: <nil>
Dec 30 06:13:05.272783788 (590h44m5.143866834s) [noise=true , address=77.90.31.14:4688 , piece_id=XJU7EHH2573FOVLOSLGAA46JEZG2AIKLVKZOXRBYYM7X56E22RGA, seg_pos={0 0}, upload=517864]: Done uploading piece. err: <nil>
Dec 30 06:13:05.667537083 (590h44m5.538620128s) [noise=true , address=145.79.227.43:4189 , piece_id=FZK7EWS2AKZZZL4B45TAR3SVPJ2UYFJGR35HIIPU3FUEWHRSX36A, seg_pos={0 0}, upload=517864]: Done uploading piece. err: <nil>
Dec 30 06:13:05.691236510 (590h44m5.562319555s) [noise=true , address=193.149.164.143:4699 , piece_id=MVZB6VSTMP3J44LDQDK7QXIA27AB3XLPIGYNHLZDXJODYOSTN3EQ, seg_pos={0 0}, upload=517864]: Done uploading piece. err: <nil>
Dec 30 06:13:05.712297864 (590h44m5.583380899s) [noise=true , address=145.79.227.30:4771 , piece_id=U3PVHRB7KCA76GXAO3SJJOED3NUEJ3B3TXXQE6SX4DJEM6M5O4RA, seg_pos={0 0}, upload=517864]: Done uploading piece. err: <nil>
Dec 30 06:13:05.720879355 (590h44m5.591962401s) [noise=true , address=145.79.227.40:4082 , piece_id=4BS3GCGUFDSZLFXPPXAPT2524TJRLZT6GMH6Y76HLIFQDHXNPKBA, seg_pos={0 0}, upload=517864]: Done uploading piece. err: <nil>
Dec 30 06:13:05.954725096 (590h44m5.825808141s) [noise=true , address=145.79.227.33:4898 , piece_id=4J7VGL3KNWH3GZ4ON4KXZLLCZLBUGHROKIIB7WGY63RHH5IPF2IA, seg_pos={0 0}, upload=517864]: Done uploading piece. err: <nil>
Dec 30 06:13:05.954752869 (590h44m5.825835914s) [seg_pos={0 0}, upload=517864]: Segment reached optimal threshold of 14 pieces.
Dec 30 06:13:05.955016099 (590h44m5.826099144s) [noise=true , address=64.49.13.169:4709 , piece_id=4QGR676MN3UCO7VOCOLCSMY2EEVDN4E2BQEZ6T73BBBUFEY7F7CQ, seg_pos={0 0}, upload=517864]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qFj8Hrgxx3ZMjo1tzyJ5nqjGmF3ahUuisy2rXWLztxDRVRR4K): protocol: write tcp 188.166.225.170:49652->64.49.13.169:4709: use of closed network connection; write tcp 188.166.225.170:49652->64.49.13.169:4709: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49652->64.49.13.169:4709: use of closed network connection
Dec 30 06:13:05.955027190 (590h44m5.826110225s) [noise=true , address=64.49.13.169:4709 , piece_id=4QGR676MN3UCO7VOCOLCSMY2EEVDN4E2BQEZ6T73BBBUFEY7F7CQ, seg_pos={0 0}, upload=517864]: Long tail cancellation due to optimal threshold
Dec 30 06:13:05.955054612 (590h44m5.826137658s) [noise=true , address=45.91.116.11:4139 , piece_id=CKEVNQMM5MAMUE4XHWYN6SD4MWZB5CMMYAB7JDZ3C43NNMDSJVCA, seg_pos={0 0}, upload=517864]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Tz3HEHACmHU9Gp2W2vHVuF5PXNmdsa5N8nYhwt52HCTjggqRD): protocol: expected piece hash; context canceled; EOF
Dec 30 06:13:05.955062788 (590h44m5.826145833s) [noise=true , address=45.91.116.11:4139 , piece_id=CKEVNQMM5MAMUE4XHWYN6SD4MWZB5CMMYAB7JDZ3C43NNMDSJVCA, seg_pos={0 0}, upload=517864]: Long tail cancellation due to optimal threshold
Dec 30 06:13:05.955166545 (590h44m5.82624959s) [noise=true , address=77.90.31.18:4833 , piece_id=5OHYRK74NRJAS6ENJIEEP3NXETW2GIOHESUCHHLIUCOO5E3TENPQ, seg_pos={0 0}, upload=517864]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FbVbo8kGy1aaFyCk2noiXNUJHtosFJHFykBMRecGd5Kj5gjEp): protocol: expected piece hash; context canceled; EOF
Dec 30 06:13:05.955172297 (590h44m5.826255342s) [noise=true , address=77.90.31.18:4833 , piece_id=5OHYRK74NRJAS6ENJIEEP3NXETW2GIOHESUCHHLIUCOO5E3TENPQ, seg_pos={0 0}, upload=517864]: Long tail cancellation due to optimal threshold
Dec 30 06:13:05.955273479 (590h44m5.826356524s) [noise=true , address=64.49.13.169:4716 , piece_id=3HZZKO6XDXO2JQCUOIXIWUKWRQ3WESRYW3TPRQWTFTM6UYUE4VOQ, seg_pos={0 0}, upload=517864]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:49808->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:49808->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49808->64.49.13.169:4716: use of closed network connection
Dec 30 06:13:05.955280313 (590h44m5.826363348s) [noise=true , address=64.49.13.169:4716 , piece_id=3HZZKO6XDXO2JQCUOIXIWUKWRQ3WESRYW3TPRQWTFTM6UYUE4VOQ, seg_pos={0 0}, upload=517864]: Long tail cancellation due to optimal threshold
Dec 30 06:13:05.955310550 (590h44m5.826393596s) [seg_pos={0 0}, upload=517864]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 06:13:05.955381786 (590h44m5.826464821s) [upload=517864]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:13:05.955386705 (590h44m5.82646974s) [upload=517864]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:13:06.243506482 (590h44m6.114589527s) [upload=517864]: Done uploading segments...
Dec 30 06:13:06.243537802 (590h44m6.114620837s) [upload=517864]: Upload finished. err: <nil>
Duration: 42.433049547s
core
uplink
us-select-1
us1-placement-20
|