Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
25 out of 60 is failed (threshold=6) [...XXXX....XXX.X...X....X...XX.X..XXX....X..XX.X...X.X.XX..X] duration:
5.000604613s trace:
https://jaeger.dev.storj.io/trace/32845f32250d5628
FAIL (2025-12-28 06:01:57 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000625974s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [............X.........X...X....................X............] duration:
47.9027151s trace:
https://jaeger.dev.storj.io/trace/35a88c4b3a3d78b6
FAIL (2025-12-28 06:01:41 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH, Piece ID: HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA) context deadline exceeded; piecestore: (Node ID: 1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, Piece ID: GY4LXIIIB7SVWA2WRAEHJVB6HLN6XQYISF7ZPVE4KA42C23MEQ4A) context deadline exceeded; piecestore: (Node ID: 1RQ3vtArvKAvaMJTVa9VDDVdWR5oBuZ7gaPB3pzc1MouqBZhWu, Piece ID: FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ) context deadline exceeded; piecestore: (Node ID: 127KQ2X6VXddEeFRMau7MVGwtTsUmFsm8hTYDR2A6B4A1oVKfyo, Piece ID: V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ) context deadline exceeded
======LOGS======
) [noise=true , address=145.79.227.44:4209 , piece_id=JRNDKXGHFY5OJYSY4ESURYMK5BNRDWY6INLHOWEK55KHMNRJO4AQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.655023976 (542h19m51.526107021s) [noise=true , address=77.90.31.13:4647 , piece_id=SOPU7ZUZQXRLU6TUL4L3ODX7U7WFAKUDOSK5UO66WMAIANU6PVUA, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.655676448 (542h19m51.526759493s) [noise=true , address=45.91.116.8:4059 , piece_id=OSA2YK4AH2DPQBNSJDFT3PLTGPJM6N7EOB7X27ZP2WM52HWSHT4Q, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.656204463 (542h19m51.527287509s) [noise=true , address=145.79.227.49:4370 , piece_id=QDMDBC3576X7ZYGUAC2DLJKWBNVOLSROCMA4UNFDZ47TTADILUAQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.656699927 (542h19m51.527782972s) [noise=true , address=45.91.116.9:4290 , piece_id=MP5DAA5UGHWQ3BA5R7GBWNIV2B4JYCL5WUSV2PY7GTO4PNLH2F2Q, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.657183808 (542h19m51.528266883s) [noise=true , address=77.90.31.18:4827 , piece_id=V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.657682177 (542h19m51.528765222s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.658706768 (542h19m51.529789813s) [noise=true , address=145.79.227.33:4876 , piece_id=E5QQDATADJON5NOOQ6X37GFKUQCFUKJRGK6G2EURBSUJVLF72Y7A, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.660113707 (542h19m51.531196752s) [noise=true , address=145.79.227.34:4915 , piece_id=MR5WOODFPXH57I6O2JCMC3EE5OWXF23CFLLATZXFP5XK6YPWDZJA, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.661356113 (542h19m51.532439158s) [noise=true , address=77.90.31.15:4727 , piece_id=3ACH3RUAR5KUXZU23AL3OQSGIZQOU5MFOFRN3JUYLQJBJCA4JOZQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.664431910 (542h19m51.535514955s) [noise=true , address=45.91.116.6:4998 , piece_id=FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.666377204 (542h19m51.537460249s) [noise=true , address=145.79.227.40:4067 , piece_id=36Q7DHHN5ELVD3BYSNUGJ7WYXNGIDKERTOPRAZJNJ7YQTOUPWEYQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.666754502 (542h19m51.537837548s) [noise=true , address=77.90.31.12:4594 , piece_id=DNXKJQOUHLAWMOCR35THP5CKCB6LZBW4GMJU6O6KKVSIJ3ALLONQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.668451284 (542h19m51.539534319s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.671076552 (542h19m51.542159587s) [noise=true , address=77.90.31.11:4560 , piece_id=HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.674568571 (542h19m51.545651626s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:52.998088476 (542h19m52.869171521s) [upload=475588]: Next returned nil segment
Dec 28 05:48:52.998103846 (542h19m52.869186881s) [upload=475588]: Waiting for error group managing segments...
Dec 28 05:48:53.543654582 (542h19m53.414737627s) [noise=true , address=145.79.227.40:4067 , piece_id=36Q7DHHN5ELVD3BYSNUGJ7WYXNGIDKERTOPRAZJNJ7YQTOUPWEYQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.543744133 (542h19m53.414827178s) [noise=true , address=145.79.227.49:4370 , piece_id=QDMDBC3576X7ZYGUAC2DLJKWBNVOLSROCMA4UNFDZ47TTADILUAQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.556254274 (542h19m53.427337309s) [noise=true , address=145.79.227.34:4915 , piece_id=MR5WOODFPXH57I6O2JCMC3EE5OWXF23CFLLATZXFP5XK6YPWDZJA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.810781113 (542h19m53.681864158s) [noise=true , address=77.90.31.13:4647 , piece_id=SOPU7ZUZQXRLU6TUL4L3ODX7U7WFAKUDOSK5UO66WMAIANU6PVUA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.814762255 (542h19m53.6858453s) [noise=true , address=77.90.31.16:4756 , piece_id=GY4LXIIIB7SVWA2WRAEHJVB6HLN6XQYISF7ZPVE4KA42C23MEQ4A, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.816966662 (542h19m53.688049707s) [noise=true , address=77.90.31.15:4727 , piece_id=3ACH3RUAR5KUXZU23AL3OQSGIZQOU5MFOFRN3JUYLQJBJCA4JOZQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.895077718 (542h19m53.766160783s) [noise=true , address=45.91.116.9:4290 , piece_id=MP5DAA5UGHWQ3BA5R7GBWNIV2B4JYCL5WUSV2PY7GTO4PNLH2F2Q, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.906586933 (542h19m53.777669978s) [noise=true , address=45.91.116.8:4059 , piece_id=OSA2YK4AH2DPQBNSJDFT3PLTGPJM6N7EOB7X27ZP2WM52HWSHT4Q, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.930785842 (542h19m53.801868877s) [noise=true , address=145.79.227.33:4876 , piece_id=E5QQDATADJON5NOOQ6X37GFKUQCFUKJRGK6G2EURBSUJVLF72Y7A, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.087446700 (542h19m53.958529735s) [noise=true , address=45.91.116.6:4998 , piece_id=FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.129981903 (542h19m54.001064938s) [noise=true , address=77.90.31.11:4560 , piece_id=HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.264740966 (542h19m54.135824011s) [noise=true , address=77.90.31.12:4594 , piece_id=DNXKJQOUHLAWMOCR35THP5CKCB6LZBW4GMJU6O6KKVSIJ3ALLONQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.368348401 (542h19m54.239431446s) [noise=true , address=77.90.31.18:4827 , piece_id=V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.437713885 (542h19m54.30879693s) [noise=true , address=145.79.227.44:4209 , piece_id=JRNDKXGHFY5OJYSY4ESURYMK5BNRDWY6INLHOWEK55KHMNRJO4AQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.437828132 (542h19m54.308911187s) [seg_pos={0 0}, upload=475588]: Segment reached optimal threshold of 14 pieces.
Dec 28 05:48:54.438064122 (542h19m54.309147167s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF
Dec 28 05:48:54.438074481 (542h19m54.309157546s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold
Dec 28 05:48:54.438099619 (542h19m54.309182664s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F4fJAGJM2ZPRt7vJH2cZmEbYeujmxcfi7aPZ2g9VPTNYBYRAi): protocol: expected piece hash; context canceled; EOF
Dec 28 05:48:54.438108586 (542h19m54.309191631s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold
Dec 28 05:48:54.438243403 (542h19m54.309326458s) [noise=true , address=77.90.31.19:4876 , piece_id=PUIQX7IZSNPKZTZTYH32KTFUM7K53KWZASLPT6JHAPMOB2JCXPGA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12H3UNBGeAufYSdVvEgYkv2QWeFtjKMvHvYGeqGeBvjEHqDDjBB): protocol: expected piece hash; context canceled; EOF
Dec 28 05:48:54.438252169 (542h19m54.309335214s) [noise=true , address=77.90.31.19:4876 , piece_id=PUIQX7IZSNPKZTZTYH32KTFUM7K53KWZASLPT6JHAPMOB2JCXPGA, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold
Dec 28 05:48:54.438391024 (542h19m54.309474069s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection
Dec 28 05:48:54.438398588 (542h19m54.309481633s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold
Dec 28 05:48:54.438699081 (542h19m54.309782126s) [seg_pos={0 0}, upload=475588]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 05:48:54.438794753 (542h19m54.309877798s) [upload=475588]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 05:48:54.438801255 (542h19m54.30988429s) [upload=475588]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 05:48:54.728435695 (542h19m54.59951876s) [upload=475588]: Done uploading segments...
Dec 28 05:48:54.728457938 (542h19m54.599540983s) [upload=475588]: Upload finished. err: <nil>
Duration: 47.902732493s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
24 out of 60 is failed (threshold=6) [....XXXX....XXX.X...X....X...XX.X..XXX....X..XX.X...X.X.XX..] duration:
3.455633105s trace:
https://jaeger.dev.storj.io/trace/109d6b114656b8e0
FAIL (2025-12-28 06:01:37 UTC)
orbiter warning: context deadline exceeded
Duration: 3.455738396s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
18 out of 60 is failed (threshold=3) [.XX..XX.X...X.X.XX....X...X.X..X.....X........X...X..X....X.] duration:
6.074050061s trace:
https://jaeger.dev.storj.io/trace/4ac9ee181cdda84d
FAIL (2025-12-28 06:01:30 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 28 06:00:18.199082851 (542h31m18.070165896s) [upload=475754]: Starting upload
Dec 28 06:00:18.199187531 (542h31m18.070270586s) [upload=475754]: Done starting upload
Dec 28 06:00:18.199244189 (542h31m18.070327234s) [upload=475754]: Uploading segments...
Dec 28 06:00:18.199483754 (542h31m18.070566809s) [upload=475754]: Got next segment. Inline: false
Dec 28 06:00:18.199532007 (542h31m18.070615062s) [upload=475754]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 06:00:18.199538098 (542h31m18.070621143s) [upload=475754]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 06:00:18.447252205 (542h31m18.318335291s) [seg_pos={0 0}, upload=475754]: Begin upload segment...
Dec 28 06:00:18.447314845 (542h31m18.31839789s) [seg_pos={0 0}, upload=475754]: Done begin upload segment.
Dec 28 06:00:18.447356885 (542h31m18.31843994s) [noise=true , address=145.79.227.14:4837 , piece_id=Q2447ANKOBRMSXZ2JKG3EHZ7USKLJC55VRLH4HPDTMUM6JEW54UQ, seg_pos={0 0}, upload=475754]: Uploading piece...
Dec 28 06:00:18.447417611 (542h31m18.318500656s) [noise=true , address=145.79.227.10:4401 , piece_id=DCCPIMXMWTLXX2AFUE26W7QZOHYHKILMA7P7NDFBVMBJ54HPXKUA, seg_pos={0 0}, upload=475754]: Uploading piece...
Dec 28 06:00:18.447967548 (542h31m18.319050603s) [noise=true , address=145.79.227.23:4547 , piece_id=CYDVYAIVDMYTGOB35F6S4XO46FI4R4U3YGQJIF7GRYTEVOIKZYBA, seg_pos={0 0}, upload=475754]: Uploading piece...
Dec 28 06:00:18.448424007 (542h31m18.319507052s) [noise=true , address=145.79.227.48:4303 , piece_id=6BQZYSUCK6OQRUL43CHDJYR6B7AGWLTNSO3GOFIQS4TRGNZB5ERQ, seg_pos={0 0}, upload=475754]: Uploading piece...
Dec 28 06:00:18.448870908 (542h31m18.319953963s) [noise=true , address=145.79.227.51:4410 , piece_id=TLH35REAIL5IBLQ3PIWSTZU2D4NHLLAG5YE33IK6ZVYLZ4DVNHFQ, seg_pos={0 0}, upload=475754]: Uploading piece...
Dec 28 06:00:18.449318280 (542h31m18.320401315s) [noise=true , address=145.79.227.6:4258 , piece_id=RRQVBLUJYIGPBMFIWM3ONZAFBX5MQHQLOUZBXUOZS7GVC5EZLPUQ, seg_pos={0 0}, upload=475754]: Uploading piece...
Dec 28 06:00:18.449862005 (542h31m18.32094505s) [noise=true , address=145.79.227.29:4942 , piece_id=2MYNGVWE4NBR7WEVYYXOQO4TLRSV4RBKRES3ISECZWNCRPO7IWCA, seg_pos={0 0}, upload=475754]: Uploading piece...
Dec 28 06:00:18.450349313 (542h31m18.321432368s) [noise=true , address=145.79.227.31:4798 , piece_id=CRIIZA7TNBDBUFQHE3SINOHSQPO5PCWALZ523DJCXMBIH6EWYJZQ, seg_pos={0 0}, upload=475754]: Uploading piece...
Dec 28 06:00:18.451585798 (542h31m18.322668843s) [noise=true , address=145.79.227.42:4123 , piece_id=FJ3HHVANTODHV7IIAR2R5KKI5W4L4EEZ3LWH77VNAGK2PDGWNEJQ, seg_pos={0 0}, upload=475754]: Uploading piece...
Dec 28 06:00:18.453153283 (542h31m18.324236328s) [noise=true , address=145.79.227.52:4446 , piece_id=2EP3NBVB24GKTIEUQRXUQIHXR6AREZKZ5E24LHFX7EMIQ3XIGOVQ, seg_pos={0 0}, upload=475754]: Uploading piece...
Dec 28 06:00:18.453498540 (542h31m18.324581595s) [noise=true , address=77.90.31.11:4556 , piece_id=A5SYN3VKFPE3FA4C37OOPLXHWGRSLSV7BMDTPA5L6ZWXE5ZMIVFQ, seg_pos={0 0}, upload=475754]: Uploading piece...
Dec 28 06:00:18.455381105 (542h31m18.32646415s) [noise=true , address=45.91.116.6:4975 , piece_id=AGMFUF2YGZJ7VLBWFIXVHINFXYDGXQCB33LF7MWZ26GO2M3VSDZA, seg_pos={0 0}, upload=475754]: Uploading piece...
Dec 28 06:00:18.456729122 (542h31m18.327812167s) [noise=true , address=77.90.31.10:4519 , piece_id=ARPVWCN6MRQOMDN23XO2EE27ZNKAAUMEW7JKJUSQN43VXBNWFHCQ, seg_pos={0 0}, upload=475754]: Uploading piece...
Dec 28 06:00:18.458694554 (542h31m18.329777609s) [noise=true , address=145.79.227.47:4266 , piece_id=3NU2IQXHPGA3533L6FVPZX7HHKBVZVAFHAINPPDV36FYLEJYTM7A, seg_pos={0 0}, upload=475754]: Uploading piece...
Dec 28 06:00:18.460367980 (542h31m18.331451025s) [noise=true , address=145.79.227.46:4601 , piece_id=CWE7VLAX6PRB7PLKLQYDHNYFJNMEMCTPVXGYOFQ44NU4MIGKDODA, seg_pos={0 0}, upload=475754]: Uploading piece...
Dec 28 06:00:18.462359562 (542h31m18.333442618s) [noise=true , address=145.79.227.17:4971 , piece_id=W373IZA4G2AZRZHBBQZWZJKMYHB5K3LDHDBHXSZPHZ34FDFRZW7Q, seg_pos={0 0}, upload=475754]: Uploading piece...
Dec 28 06:00:20.081111091 (542h31m19.952194136s) [upload=475754]: Next returned nil segment
Dec 28 06:00:20.081130489 (542h31m19.952213554s) [upload=475754]: Waiting for error group managing segments...
Dec 28 06:00:20.480478382 (542h31m20.351561427s) [noise=true , address=145.79.227.14:4837 , piece_id=Q2447ANKOBRMSXZ2JKG3EHZ7USKLJC55VRLH4HPDTMUM6JEW54UQ, seg_pos={0 0}, upload=475754]: Done uploading piece. err: <nil>
Dec 28 06:00:20.504137433 (542h31m20.375220478s) [noise=true , address=145.79.227.29:4942 , piece_id=2MYNGVWE4NBR7WEVYYXOQO4TLRSV4RBKRES3ISECZWNCRPO7IWCA, seg_pos={0 0}, upload=475754]: Done uploading piece. err: <nil>
Dec 28 06:00:20.506366428 (542h31m20.377449473s) [noise=true , address=145.79.227.52:4446 , piece_id=2EP3NBVB24GKTIEUQRXUQIHXR6AREZKZ5E24LHFX7EMIQ3XIGOVQ, seg_pos={0 0}, upload=475754]: Done uploading piece. err: <nil>
Dec 28 06:00:20.654359050 (542h31m20.525442095s) [noise=true , address=77.90.31.11:4556 , piece_id=A5SYN3VKFPE3FA4C37OOPLXHWGRSLSV7BMDTPA5L6ZWXE5ZMIVFQ, seg_pos={0 0}, upload=475754]: Done uploading piece. err: <nil>
Dec 28 06:00:20.681173609 (542h31m20.552256654s) [noise=true , address=77.90.31.10:4519 , piece_id=ARPVWCN6MRQOMDN23XO2EE27ZNKAAUMEW7JKJUSQN43VXBNWFHCQ, seg_pos={0 0}, upload=475754]: Done uploading piece. err: <nil>
Dec 28 06:00:20.876961051 (542h31m20.748044096s) [noise=true , address=145.79.227.42:4123 , piece_id=FJ3HHVANTODHV7IIAR2R5KKI5W4L4EEZ3LWH77VNAGK2PDGWNEJQ, seg_pos={0 0}, upload=475754]: Done uploading piece. err: <nil>
Dec 28 06:00:20.888147532 (542h31m20.759230577s) [noise=true , address=145.79.227.48:4303 , piece_id=6BQZYSUCK6OQRUL43CHDJYR6B7AGWLTNSO3GOFIQS4TRGNZB5ERQ, seg_pos={0 0}, upload=475754]: Done uploading piece. err: <nil>
Dec 28 06:00:20.930573327 (542h31m20.801656372s) [noise=true , address=145.79.227.47:4266 , piece_id=3NU2IQXHPGA3533L6FVPZX7HHKBVZVAFHAINPPDV36FYLEJYTM7A, seg_pos={0 0}, upload=475754]: Done uploading piece. err: <nil>
Dec 28 06:00:20.956157904 (542h31m20.827240949s) [noise=true , address=45.91.116.6:4975 , piece_id=AGMFUF2YGZJ7VLBWFIXVHINFXYDGXQCB33LF7MWZ26GO2M3VSDZA, seg_pos={0 0}, upload=475754]: Done uploading piece. err: <nil>
Dec 28 06:00:20.975159186 (542h31m20.846242231s) [noise=true , address=145.79.227.46:4601 , piece_id=CWE7VLAX6PRB7PLKLQYDHNYFJNMEMCTPVXGYOFQ44NU4MIGKDODA, seg_pos={0 0}, upload=475754]: Done uploading piece. err: <nil>
Dec 28 06:00:20.999055348 (542h31m20.870138403s) [noise=true , address=145.79.227.51:4410 , piece_id=TLH35REAIL5IBLQ3PIWSTZU2D4NHLLAG5YE33IK6ZVYLZ4DVNHFQ, seg_pos={0 0}, upload=475754]: Done uploading piece. err: <nil>
Dec 28 06:00:21.130954967 (542h31m21.002038012s) [noise=true , address=145.79.227.10:4401 , piece_id=DCCPIMXMWTLXX2AFUE26W7QZOHYHKILMA7P7NDFBVMBJ54HPXKUA, seg_pos={0 0}, upload=475754]: Done uploading piece. err: <nil>
Dec 28 06:00:47.499134946 (542h31m47.370217981s) [noise=true , address=145.79.227.31:4798 , piece_id=CRIIZA7TNBDBUFQHE3SINOHSQPO5PCWALZ523DJCXMBIH6EWYJZQ, seg_pos={0 0}, upload=475754]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:41948->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:41948->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41948->145.79.227.31:4798: use of closed network connection
Dec 28 06:00:47.499152230 (542h31m47.370235265s) [noise=true , address=145.79.227.23:4547 , piece_id=CYDVYAIVDMYTGOB35F6S4XO46FI4R4U3YGQJIF7GRYTEVOIKZYBA, seg_pos={0 0}, upload=475754]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: write tcp 188.166.225.170:55230->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:55230->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55230->145.79.227.23:4547: use of closed network connection
Dec 28 06:00:47.499234205 (542h31m47.370317251s) [noise=true , address=145.79.227.17:4971 , piece_id=W373IZA4G2AZRZHBBQZWZJKMYHB5K3LDHDBHXSZPHZ34FDFRZW7Q, seg_pos={0 0}, upload=475754]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: write tcp 188.166.225.170:44222->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:44222->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44222->145.79.227.17:4971: use of closed network connection
Dec 28 06:00:47.499357972 (542h31m47.370441017s) [noise=true , address=145.79.227.6:4258 , piece_id=RRQVBLUJYIGPBMFIWM3ONZAFBX5MQHQLOUZBXUOZS7GVC5EZLPUQ, seg_pos={0 0}, upload=475754]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 188.166.225.170:58852->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:58852->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58852->145.79.227.6:4258: use of closed network connection
Dec 28 06:00:47.499398519 (542h31m47.370481564s) [seg_pos={0 0}, upload=475754]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 28 06:00:47.747950843 (542h31m47.619033888s) [upload=475754]: Done uploading segments...
Dec 28 06:00:47.747997482 (542h31m47.619080527s) [upload=475754]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 6.074087172s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [.............X.........X...X....................X...........] duration:
46.021923371s trace:
https://jaeger.dev.storj.io/trace/35a88c4b3a3d78b6
FAIL (2025-12-28 06:00:53 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH, Piece ID: HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA) context deadline exceeded; piecestore: (Node ID: 1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, Piece ID: GY4LXIIIB7SVWA2WRAEHJVB6HLN6XQYISF7ZPVE4KA42C23MEQ4A) context deadline exceeded; piecestore: (Node ID: 1RQ3vtArvKAvaMJTVa9VDDVdWR5oBuZ7gaPB3pzc1MouqBZhWu, Piece ID: FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ) context deadline exceeded; piecestore: (Node ID: 127KQ2X6VXddEeFRMau7MVGwtTsUmFsm8hTYDR2A6B4A1oVKfyo, Piece ID: V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ) context deadline exceeded
======LOGS======
) [noise=true , address=145.79.227.44:4209 , piece_id=JRNDKXGHFY5OJYSY4ESURYMK5BNRDWY6INLHOWEK55KHMNRJO4AQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.655023976 (542h19m51.526107021s) [noise=true , address=77.90.31.13:4647 , piece_id=SOPU7ZUZQXRLU6TUL4L3ODX7U7WFAKUDOSK5UO66WMAIANU6PVUA, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.655676448 (542h19m51.526759493s) [noise=true , address=45.91.116.8:4059 , piece_id=OSA2YK4AH2DPQBNSJDFT3PLTGPJM6N7EOB7X27ZP2WM52HWSHT4Q, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.656204463 (542h19m51.527287509s) [noise=true , address=145.79.227.49:4370 , piece_id=QDMDBC3576X7ZYGUAC2DLJKWBNVOLSROCMA4UNFDZ47TTADILUAQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.656699927 (542h19m51.527782972s) [noise=true , address=45.91.116.9:4290 , piece_id=MP5DAA5UGHWQ3BA5R7GBWNIV2B4JYCL5WUSV2PY7GTO4PNLH2F2Q, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.657183808 (542h19m51.528266883s) [noise=true , address=77.90.31.18:4827 , piece_id=V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.657682177 (542h19m51.528765222s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.658706768 (542h19m51.529789813s) [noise=true , address=145.79.227.33:4876 , piece_id=E5QQDATADJON5NOOQ6X37GFKUQCFUKJRGK6G2EURBSUJVLF72Y7A, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.660113707 (542h19m51.531196752s) [noise=true , address=145.79.227.34:4915 , piece_id=MR5WOODFPXH57I6O2JCMC3EE5OWXF23CFLLATZXFP5XK6YPWDZJA, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.661356113 (542h19m51.532439158s) [noise=true , address=77.90.31.15:4727 , piece_id=3ACH3RUAR5KUXZU23AL3OQSGIZQOU5MFOFRN3JUYLQJBJCA4JOZQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.664431910 (542h19m51.535514955s) [noise=true , address=45.91.116.6:4998 , piece_id=FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.666377204 (542h19m51.537460249s) [noise=true , address=145.79.227.40:4067 , piece_id=36Q7DHHN5ELVD3BYSNUGJ7WYXNGIDKERTOPRAZJNJ7YQTOUPWEYQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.666754502 (542h19m51.537837548s) [noise=true , address=77.90.31.12:4594 , piece_id=DNXKJQOUHLAWMOCR35THP5CKCB6LZBW4GMJU6O6KKVSIJ3ALLONQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.668451284 (542h19m51.539534319s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.671076552 (542h19m51.542159587s) [noise=true , address=77.90.31.11:4560 , piece_id=HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.674568571 (542h19m51.545651626s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:52.998088476 (542h19m52.869171521s) [upload=475588]: Next returned nil segment
Dec 28 05:48:52.998103846 (542h19m52.869186881s) [upload=475588]: Waiting for error group managing segments...
Dec 28 05:48:53.543654582 (542h19m53.414737627s) [noise=true , address=145.79.227.40:4067 , piece_id=36Q7DHHN5ELVD3BYSNUGJ7WYXNGIDKERTOPRAZJNJ7YQTOUPWEYQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.543744133 (542h19m53.414827178s) [noise=true , address=145.79.227.49:4370 , piece_id=QDMDBC3576X7ZYGUAC2DLJKWBNVOLSROCMA4UNFDZ47TTADILUAQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.556254274 (542h19m53.427337309s) [noise=true , address=145.79.227.34:4915 , piece_id=MR5WOODFPXH57I6O2JCMC3EE5OWXF23CFLLATZXFP5XK6YPWDZJA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.810781113 (542h19m53.681864158s) [noise=true , address=77.90.31.13:4647 , piece_id=SOPU7ZUZQXRLU6TUL4L3ODX7U7WFAKUDOSK5UO66WMAIANU6PVUA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.814762255 (542h19m53.6858453s) [noise=true , address=77.90.31.16:4756 , piece_id=GY4LXIIIB7SVWA2WRAEHJVB6HLN6XQYISF7ZPVE4KA42C23MEQ4A, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.816966662 (542h19m53.688049707s) [noise=true , address=77.90.31.15:4727 , piece_id=3ACH3RUAR5KUXZU23AL3OQSGIZQOU5MFOFRN3JUYLQJBJCA4JOZQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.895077718 (542h19m53.766160783s) [noise=true , address=45.91.116.9:4290 , piece_id=MP5DAA5UGHWQ3BA5R7GBWNIV2B4JYCL5WUSV2PY7GTO4PNLH2F2Q, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.906586933 (542h19m53.777669978s) [noise=true , address=45.91.116.8:4059 , piece_id=OSA2YK4AH2DPQBNSJDFT3PLTGPJM6N7EOB7X27ZP2WM52HWSHT4Q, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.930785842 (542h19m53.801868877s) [noise=true , address=145.79.227.33:4876 , piece_id=E5QQDATADJON5NOOQ6X37GFKUQCFUKJRGK6G2EURBSUJVLF72Y7A, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.087446700 (542h19m53.958529735s) [noise=true , address=45.91.116.6:4998 , piece_id=FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.129981903 (542h19m54.001064938s) [noise=true , address=77.90.31.11:4560 , piece_id=HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.264740966 (542h19m54.135824011s) [noise=true , address=77.90.31.12:4594 , piece_id=DNXKJQOUHLAWMOCR35THP5CKCB6LZBW4GMJU6O6KKVSIJ3ALLONQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.368348401 (542h19m54.239431446s) [noise=true , address=77.90.31.18:4827 , piece_id=V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.437713885 (542h19m54.30879693s) [noise=true , address=145.79.227.44:4209 , piece_id=JRNDKXGHFY5OJYSY4ESURYMK5BNRDWY6INLHOWEK55KHMNRJO4AQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.437828132 (542h19m54.308911187s) [seg_pos={0 0}, upload=475588]: Segment reached optimal threshold of 14 pieces.
Dec 28 05:48:54.438064122 (542h19m54.309147167s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF
Dec 28 05:48:54.438074481 (542h19m54.309157546s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold
Dec 28 05:48:54.438099619 (542h19m54.309182664s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F4fJAGJM2ZPRt7vJH2cZmEbYeujmxcfi7aPZ2g9VPTNYBYRAi): protocol: expected piece hash; context canceled; EOF
Dec 28 05:48:54.438108586 (542h19m54.309191631s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold
Dec 28 05:48:54.438243403 (542h19m54.309326458s) [noise=true , address=77.90.31.19:4876 , piece_id=PUIQX7IZSNPKZTZTYH32KTFUM7K53KWZASLPT6JHAPMOB2JCXPGA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12H3UNBGeAufYSdVvEgYkv2QWeFtjKMvHvYGeqGeBvjEHqDDjBB): protocol: expected piece hash; context canceled; EOF
Dec 28 05:48:54.438252169 (542h19m54.309335214s) [noise=true , address=77.90.31.19:4876 , piece_id=PUIQX7IZSNPKZTZTYH32KTFUM7K53KWZASLPT6JHAPMOB2JCXPGA, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold
Dec 28 05:48:54.438391024 (542h19m54.309474069s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection
Dec 28 05:48:54.438398588 (542h19m54.309481633s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold
Dec 28 05:48:54.438699081 (542h19m54.309782126s) [seg_pos={0 0}, upload=475588]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 05:48:54.438794753 (542h19m54.309877798s) [upload=475588]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 05:48:54.438801255 (542h19m54.30988429s) [upload=475588]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 05:48:54.728435695 (542h19m54.59951876s) [upload=475588]: Done uploading segments...
Dec 28 05:48:54.728457938 (542h19m54.599540983s) [upload=475588]: Upload finished. err: <nil>
Duration: 46.021967335s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [X.XX..XX.X...X.X.XX....X...X.X..X.....X........X...X..X....X] duration:
30.251020596s trace:
https://jaeger.dev.storj.io/trace/4ac9ee181cdda84d
FAIL (2025-12-28 06:00:47 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 28 06:00:18.199082851 (542h31m18.070165896s) [upload=475754]: Starting upload
Dec 28 06:00:18.199187531 (542h31m18.070270586s) [upload=475754]: Done starting upload
Dec 28 06:00:18.199244189 (542h31m18.070327234s) [upload=475754]: Uploading segments...
Dec 28 06:00:18.199483754 (542h31m18.070566809s) [upload=475754]: Got next segment. Inline: false
Dec 28 06:00:18.199532007 (542h31m18.070615062s) [upload=475754]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 06:00:18.199538098 (542h31m18.070621143s) [upload=475754]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 06:00:18.447252205 (542h31m18.318335291s) [seg_pos={0 0}, upload=475754]: Begin upload segment...
Dec 28 06:00:18.447314845 (542h31m18.31839789s) [seg_pos={0 0}, upload=475754]: Done begin upload segment.
Dec 28 06:00:18.447356885 (542h31m18.31843994s) [noise=true , address=145.79.227.14:4837 , piece_id=Q2447ANKOBRMSXZ2JKG3EHZ7USKLJC55VRLH4HPDTMUM6JEW54UQ, seg_pos={0 0}, upload=475754]: Uploading piece...
Dec 28 06:00:18.447417611 (542h31m18.318500656s) [noise=true , address=145.79.227.10:4401 , piece_id=DCCPIMXMWTLXX2AFUE26W7QZOHYHKILMA7P7NDFBVMBJ54HPXKUA, seg_pos={0 0}, upload=475754]: Uploading piece...
Dec 28 06:00:18.447967548 (542h31m18.319050603s) [noise=true , address=145.79.227.23:4547 , piece_id=CYDVYAIVDMYTGOB35F6S4XO46FI4R4U3YGQJIF7GRYTEVOIKZYBA, seg_pos={0 0}, upload=475754]: Uploading piece...
Dec 28 06:00:18.448424007 (542h31m18.319507052s) [noise=true , address=145.79.227.48:4303 , piece_id=6BQZYSUCK6OQRUL43CHDJYR6B7AGWLTNSO3GOFIQS4TRGNZB5ERQ, seg_pos={0 0}, upload=475754]: Uploading piece...
Dec 28 06:00:18.448870908 (542h31m18.319953963s) [noise=true , address=145.79.227.51:4410 , piece_id=TLH35REAIL5IBLQ3PIWSTZU2D4NHLLAG5YE33IK6ZVYLZ4DVNHFQ, seg_pos={0 0}, upload=475754]: Uploading piece...
Dec 28 06:00:18.449318280 (542h31m18.320401315s) [noise=true , address=145.79.227.6:4258 , piece_id=RRQVBLUJYIGPBMFIWM3ONZAFBX5MQHQLOUZBXUOZS7GVC5EZLPUQ, seg_pos={0 0}, upload=475754]: Uploading piece...
Dec 28 06:00:18.449862005 (542h31m18.32094505s) [noise=true , address=145.79.227.29:4942 , piece_id=2MYNGVWE4NBR7WEVYYXOQO4TLRSV4RBKRES3ISECZWNCRPO7IWCA, seg_pos={0 0}, upload=475754]: Uploading piece...
Dec 28 06:00:18.450349313 (542h31m18.321432368s) [noise=true , address=145.79.227.31:4798 , piece_id=CRIIZA7TNBDBUFQHE3SINOHSQPO5PCWALZ523DJCXMBIH6EWYJZQ, seg_pos={0 0}, upload=475754]: Uploading piece...
Dec 28 06:00:18.451585798 (542h31m18.322668843s) [noise=true , address=145.79.227.42:4123 , piece_id=FJ3HHVANTODHV7IIAR2R5KKI5W4L4EEZ3LWH77VNAGK2PDGWNEJQ, seg_pos={0 0}, upload=475754]: Uploading piece...
Dec 28 06:00:18.453153283 (542h31m18.324236328s) [noise=true , address=145.79.227.52:4446 , piece_id=2EP3NBVB24GKTIEUQRXUQIHXR6AREZKZ5E24LHFX7EMIQ3XIGOVQ, seg_pos={0 0}, upload=475754]: Uploading piece...
Dec 28 06:00:18.453498540 (542h31m18.324581595s) [noise=true , address=77.90.31.11:4556 , piece_id=A5SYN3VKFPE3FA4C37OOPLXHWGRSLSV7BMDTPA5L6ZWXE5ZMIVFQ, seg_pos={0 0}, upload=475754]: Uploading piece...
Dec 28 06:00:18.455381105 (542h31m18.32646415s) [noise=true , address=45.91.116.6:4975 , piece_id=AGMFUF2YGZJ7VLBWFIXVHINFXYDGXQCB33LF7MWZ26GO2M3VSDZA, seg_pos={0 0}, upload=475754]: Uploading piece...
Dec 28 06:00:18.456729122 (542h31m18.327812167s) [noise=true , address=77.90.31.10:4519 , piece_id=ARPVWCN6MRQOMDN23XO2EE27ZNKAAUMEW7JKJUSQN43VXBNWFHCQ, seg_pos={0 0}, upload=475754]: Uploading piece...
Dec 28 06:00:18.458694554 (542h31m18.329777609s) [noise=true , address=145.79.227.47:4266 , piece_id=3NU2IQXHPGA3533L6FVPZX7HHKBVZVAFHAINPPDV36FYLEJYTM7A, seg_pos={0 0}, upload=475754]: Uploading piece...
Dec 28 06:00:18.460367980 (542h31m18.331451025s) [noise=true , address=145.79.227.46:4601 , piece_id=CWE7VLAX6PRB7PLKLQYDHNYFJNMEMCTPVXGYOFQ44NU4MIGKDODA, seg_pos={0 0}, upload=475754]: Uploading piece...
Dec 28 06:00:18.462359562 (542h31m18.333442618s) [noise=true , address=145.79.227.17:4971 , piece_id=W373IZA4G2AZRZHBBQZWZJKMYHB5K3LDHDBHXSZPHZ34FDFRZW7Q, seg_pos={0 0}, upload=475754]: Uploading piece...
Dec 28 06:00:20.081111091 (542h31m19.952194136s) [upload=475754]: Next returned nil segment
Dec 28 06:00:20.081130489 (542h31m19.952213554s) [upload=475754]: Waiting for error group managing segments...
Dec 28 06:00:20.480478382 (542h31m20.351561427s) [noise=true , address=145.79.227.14:4837 , piece_id=Q2447ANKOBRMSXZ2JKG3EHZ7USKLJC55VRLH4HPDTMUM6JEW54UQ, seg_pos={0 0}, upload=475754]: Done uploading piece. err: <nil>
Dec 28 06:00:20.504137433 (542h31m20.375220478s) [noise=true , address=145.79.227.29:4942 , piece_id=2MYNGVWE4NBR7WEVYYXOQO4TLRSV4RBKRES3ISECZWNCRPO7IWCA, seg_pos={0 0}, upload=475754]: Done uploading piece. err: <nil>
Dec 28 06:00:20.506366428 (542h31m20.377449473s) [noise=true , address=145.79.227.52:4446 , piece_id=2EP3NBVB24GKTIEUQRXUQIHXR6AREZKZ5E24LHFX7EMIQ3XIGOVQ, seg_pos={0 0}, upload=475754]: Done uploading piece. err: <nil>
Dec 28 06:00:20.654359050 (542h31m20.525442095s) [noise=true , address=77.90.31.11:4556 , piece_id=A5SYN3VKFPE3FA4C37OOPLXHWGRSLSV7BMDTPA5L6ZWXE5ZMIVFQ, seg_pos={0 0}, upload=475754]: Done uploading piece. err: <nil>
Dec 28 06:00:20.681173609 (542h31m20.552256654s) [noise=true , address=77.90.31.10:4519 , piece_id=ARPVWCN6MRQOMDN23XO2EE27ZNKAAUMEW7JKJUSQN43VXBNWFHCQ, seg_pos={0 0}, upload=475754]: Done uploading piece. err: <nil>
Dec 28 06:00:20.876961051 (542h31m20.748044096s) [noise=true , address=145.79.227.42:4123 , piece_id=FJ3HHVANTODHV7IIAR2R5KKI5W4L4EEZ3LWH77VNAGK2PDGWNEJQ, seg_pos={0 0}, upload=475754]: Done uploading piece. err: <nil>
Dec 28 06:00:20.888147532 (542h31m20.759230577s) [noise=true , address=145.79.227.48:4303 , piece_id=6BQZYSUCK6OQRUL43CHDJYR6B7AGWLTNSO3GOFIQS4TRGNZB5ERQ, seg_pos={0 0}, upload=475754]: Done uploading piece. err: <nil>
Dec 28 06:00:20.930573327 (542h31m20.801656372s) [noise=true , address=145.79.227.47:4266 , piece_id=3NU2IQXHPGA3533L6FVPZX7HHKBVZVAFHAINPPDV36FYLEJYTM7A, seg_pos={0 0}, upload=475754]: Done uploading piece. err: <nil>
Dec 28 06:00:20.956157904 (542h31m20.827240949s) [noise=true , address=45.91.116.6:4975 , piece_id=AGMFUF2YGZJ7VLBWFIXVHINFXYDGXQCB33LF7MWZ26GO2M3VSDZA, seg_pos={0 0}, upload=475754]: Done uploading piece. err: <nil>
Dec 28 06:00:20.975159186 (542h31m20.846242231s) [noise=true , address=145.79.227.46:4601 , piece_id=CWE7VLAX6PRB7PLKLQYDHNYFJNMEMCTPVXGYOFQ44NU4MIGKDODA, seg_pos={0 0}, upload=475754]: Done uploading piece. err: <nil>
Dec 28 06:00:20.999055348 (542h31m20.870138403s) [noise=true , address=145.79.227.51:4410 , piece_id=TLH35REAIL5IBLQ3PIWSTZU2D4NHLLAG5YE33IK6ZVYLZ4DVNHFQ, seg_pos={0 0}, upload=475754]: Done uploading piece. err: <nil>
Dec 28 06:00:21.130954967 (542h31m21.002038012s) [noise=true , address=145.79.227.10:4401 , piece_id=DCCPIMXMWTLXX2AFUE26W7QZOHYHKILMA7P7NDFBVMBJ54HPXKUA, seg_pos={0 0}, upload=475754]: Done uploading piece. err: <nil>
Dec 28 06:00:47.499134946 (542h31m47.370217981s) [noise=true , address=145.79.227.31:4798 , piece_id=CRIIZA7TNBDBUFQHE3SINOHSQPO5PCWALZ523DJCXMBIH6EWYJZQ, seg_pos={0 0}, upload=475754]: Done uploading piece. err: ecclient: upload failed (node:17Ski7oMzbGroPGgWRrbUiqC1NiLPBBsvYuU4pCZ53xhNKkeBZ, address:145.79.227.31:4798): protocol: write tcp 188.166.225.170:41948->145.79.227.31:4798: use of closed network connection; write tcp 188.166.225.170:41948->145.79.227.31:4798: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41948->145.79.227.31:4798: use of closed network connection
Dec 28 06:00:47.499152230 (542h31m47.370235265s) [noise=true , address=145.79.227.23:4547 , piece_id=CYDVYAIVDMYTGOB35F6S4XO46FI4R4U3YGQJIF7GRYTEVOIKZYBA, seg_pos={0 0}, upload=475754]: Done uploading piece. err: ecclient: upload failed (node:1UjJbodRCRm3E63TrzfhbjwFpCmCtfCLS34sQFi1gDJbEwJiKW, address:145.79.227.23:4547): protocol: write tcp 188.166.225.170:55230->145.79.227.23:4547: use of closed network connection; write tcp 188.166.225.170:55230->145.79.227.23:4547: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55230->145.79.227.23:4547: use of closed network connection
Dec 28 06:00:47.499234205 (542h31m47.370317251s) [noise=true , address=145.79.227.17:4971 , piece_id=W373IZA4G2AZRZHBBQZWZJKMYHB5K3LDHDBHXSZPHZ34FDFRZW7Q, seg_pos={0 0}, upload=475754]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:145.79.227.17:4971): protocol: write tcp 188.166.225.170:44222->145.79.227.17:4971: use of closed network connection; write tcp 188.166.225.170:44222->145.79.227.17:4971: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44222->145.79.227.17:4971: use of closed network connection
Dec 28 06:00:47.499357972 (542h31m47.370441017s) [noise=true , address=145.79.227.6:4258 , piece_id=RRQVBLUJYIGPBMFIWM3ONZAFBX5MQHQLOUZBXUOZS7GVC5EZLPUQ, seg_pos={0 0}, upload=475754]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: write tcp 188.166.225.170:58852->145.79.227.6:4258: use of closed network connection; write tcp 188.166.225.170:58852->145.79.227.6:4258: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58852->145.79.227.6:4258: use of closed network connection
Dec 28 06:00:47.499398519 (542h31m47.370481564s) [seg_pos={0 0}, upload=475754]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 28 06:00:47.747950843 (542h31m47.619033888s) [upload=475754]: Done uploading segments...
Dec 28 06:00:47.747997482 (542h31m47.619080527s) [upload=475754]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 30.251052586s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
25 out of 60 is failed (threshold=6) [X....XXXX....XXX.X...X....X...XX.X..XXX....X..XX.X...X.X.XX.] duration:
5.000610463s trace:
https://jaeger.dev.storj.io/trace/109d6b114656b8e0
FAIL (2025-12-28 06:00:32 UTC)
orbiter warning: context deadline exceeded
Duration: 5.000794342s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [..............X.........X...X....................X..........] duration:
34.734708992s trace:
https://jaeger.dev.storj.io/trace/35a88c4b3a3d78b6
FAIL (2025-12-28 05:59:50 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH, Piece ID: HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA) context deadline exceeded; piecestore: (Node ID: 1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, Piece ID: GY4LXIIIB7SVWA2WRAEHJVB6HLN6XQYISF7ZPVE4KA42C23MEQ4A) context deadline exceeded; piecestore: (Node ID: 1RQ3vtArvKAvaMJTVa9VDDVdWR5oBuZ7gaPB3pzc1MouqBZhWu, Piece ID: FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ) context deadline exceeded; piecestore: (Node ID: 127KQ2X6VXddEeFRMau7MVGwtTsUmFsm8hTYDR2A6B4A1oVKfyo, Piece ID: V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ) context deadline exceeded
======LOGS======
) [noise=true , address=145.79.227.44:4209 , piece_id=JRNDKXGHFY5OJYSY4ESURYMK5BNRDWY6INLHOWEK55KHMNRJO4AQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.655023976 (542h19m51.526107021s) [noise=true , address=77.90.31.13:4647 , piece_id=SOPU7ZUZQXRLU6TUL4L3ODX7U7WFAKUDOSK5UO66WMAIANU6PVUA, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.655676448 (542h19m51.526759493s) [noise=true , address=45.91.116.8:4059 , piece_id=OSA2YK4AH2DPQBNSJDFT3PLTGPJM6N7EOB7X27ZP2WM52HWSHT4Q, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.656204463 (542h19m51.527287509s) [noise=true , address=145.79.227.49:4370 , piece_id=QDMDBC3576X7ZYGUAC2DLJKWBNVOLSROCMA4UNFDZ47TTADILUAQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.656699927 (542h19m51.527782972s) [noise=true , address=45.91.116.9:4290 , piece_id=MP5DAA5UGHWQ3BA5R7GBWNIV2B4JYCL5WUSV2PY7GTO4PNLH2F2Q, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.657183808 (542h19m51.528266883s) [noise=true , address=77.90.31.18:4827 , piece_id=V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.657682177 (542h19m51.528765222s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.658706768 (542h19m51.529789813s) [noise=true , address=145.79.227.33:4876 , piece_id=E5QQDATADJON5NOOQ6X37GFKUQCFUKJRGK6G2EURBSUJVLF72Y7A, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.660113707 (542h19m51.531196752s) [noise=true , address=145.79.227.34:4915 , piece_id=MR5WOODFPXH57I6O2JCMC3EE5OWXF23CFLLATZXFP5XK6YPWDZJA, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.661356113 (542h19m51.532439158s) [noise=true , address=77.90.31.15:4727 , piece_id=3ACH3RUAR5KUXZU23AL3OQSGIZQOU5MFOFRN3JUYLQJBJCA4JOZQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.664431910 (542h19m51.535514955s) [noise=true , address=45.91.116.6:4998 , piece_id=FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.666377204 (542h19m51.537460249s) [noise=true , address=145.79.227.40:4067 , piece_id=36Q7DHHN5ELVD3BYSNUGJ7WYXNGIDKERTOPRAZJNJ7YQTOUPWEYQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.666754502 (542h19m51.537837548s) [noise=true , address=77.90.31.12:4594 , piece_id=DNXKJQOUHLAWMOCR35THP5CKCB6LZBW4GMJU6O6KKVSIJ3ALLONQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.668451284 (542h19m51.539534319s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.671076552 (542h19m51.542159587s) [noise=true , address=77.90.31.11:4560 , piece_id=HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.674568571 (542h19m51.545651626s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:52.998088476 (542h19m52.869171521s) [upload=475588]: Next returned nil segment
Dec 28 05:48:52.998103846 (542h19m52.869186881s) [upload=475588]: Waiting for error group managing segments...
Dec 28 05:48:53.543654582 (542h19m53.414737627s) [noise=true , address=145.79.227.40:4067 , piece_id=36Q7DHHN5ELVD3BYSNUGJ7WYXNGIDKERTOPRAZJNJ7YQTOUPWEYQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.543744133 (542h19m53.414827178s) [noise=true , address=145.79.227.49:4370 , piece_id=QDMDBC3576X7ZYGUAC2DLJKWBNVOLSROCMA4UNFDZ47TTADILUAQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.556254274 (542h19m53.427337309s) [noise=true , address=145.79.227.34:4915 , piece_id=MR5WOODFPXH57I6O2JCMC3EE5OWXF23CFLLATZXFP5XK6YPWDZJA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.810781113 (542h19m53.681864158s) [noise=true , address=77.90.31.13:4647 , piece_id=SOPU7ZUZQXRLU6TUL4L3ODX7U7WFAKUDOSK5UO66WMAIANU6PVUA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.814762255 (542h19m53.6858453s) [noise=true , address=77.90.31.16:4756 , piece_id=GY4LXIIIB7SVWA2WRAEHJVB6HLN6XQYISF7ZPVE4KA42C23MEQ4A, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.816966662 (542h19m53.688049707s) [noise=true , address=77.90.31.15:4727 , piece_id=3ACH3RUAR5KUXZU23AL3OQSGIZQOU5MFOFRN3JUYLQJBJCA4JOZQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.895077718 (542h19m53.766160783s) [noise=true , address=45.91.116.9:4290 , piece_id=MP5DAA5UGHWQ3BA5R7GBWNIV2B4JYCL5WUSV2PY7GTO4PNLH2F2Q, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.906586933 (542h19m53.777669978s) [noise=true , address=45.91.116.8:4059 , piece_id=OSA2YK4AH2DPQBNSJDFT3PLTGPJM6N7EOB7X27ZP2WM52HWSHT4Q, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.930785842 (542h19m53.801868877s) [noise=true , address=145.79.227.33:4876 , piece_id=E5QQDATADJON5NOOQ6X37GFKUQCFUKJRGK6G2EURBSUJVLF72Y7A, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.087446700 (542h19m53.958529735s) [noise=true , address=45.91.116.6:4998 , piece_id=FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.129981903 (542h19m54.001064938s) [noise=true , address=77.90.31.11:4560 , piece_id=HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.264740966 (542h19m54.135824011s) [noise=true , address=77.90.31.12:4594 , piece_id=DNXKJQOUHLAWMOCR35THP5CKCB6LZBW4GMJU6O6KKVSIJ3ALLONQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.368348401 (542h19m54.239431446s) [noise=true , address=77.90.31.18:4827 , piece_id=V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.437713885 (542h19m54.30879693s) [noise=true , address=145.79.227.44:4209 , piece_id=JRNDKXGHFY5OJYSY4ESURYMK5BNRDWY6INLHOWEK55KHMNRJO4AQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.437828132 (542h19m54.308911187s) [seg_pos={0 0}, upload=475588]: Segment reached optimal threshold of 14 pieces.
Dec 28 05:48:54.438064122 (542h19m54.309147167s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF
Dec 28 05:48:54.438074481 (542h19m54.309157546s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold
Dec 28 05:48:54.438099619 (542h19m54.309182664s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F4fJAGJM2ZPRt7vJH2cZmEbYeujmxcfi7aPZ2g9VPTNYBYRAi): protocol: expected piece hash; context canceled; EOF
Dec 28 05:48:54.438108586 (542h19m54.309191631s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold
Dec 28 05:48:54.438243403 (542h19m54.309326458s) [noise=true , address=77.90.31.19:4876 , piece_id=PUIQX7IZSNPKZTZTYH32KTFUM7K53KWZASLPT6JHAPMOB2JCXPGA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12H3UNBGeAufYSdVvEgYkv2QWeFtjKMvHvYGeqGeBvjEHqDDjBB): protocol: expected piece hash; context canceled; EOF
Dec 28 05:48:54.438252169 (542h19m54.309335214s) [noise=true , address=77.90.31.19:4876 , piece_id=PUIQX7IZSNPKZTZTYH32KTFUM7K53KWZASLPT6JHAPMOB2JCXPGA, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold
Dec 28 05:48:54.438391024 (542h19m54.309474069s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection
Dec 28 05:48:54.438398588 (542h19m54.309481633s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold
Dec 28 05:48:54.438699081 (542h19m54.309782126s) [seg_pos={0 0}, upload=475588]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 05:48:54.438794753 (542h19m54.309877798s) [upload=475588]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 05:48:54.438801255 (542h19m54.30988429s) [upload=475588]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 05:48:54.728435695 (542h19m54.59951876s) [upload=475588]: Done uploading segments...
Dec 28 05:48:54.728457938 (542h19m54.599540983s) [upload=475588]: Upload finished. err: <nil>
Duration: 34.734750781s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
26 out of 60 is failed (threshold=6) [XX....XXXX....XXX.X...X....X...XX.X..XXX....X..XX.X...X.X.XX] duration:
5.000998997s trace:
https://jaeger.dev.storj.io/trace/109d6b114656b8e0
FAIL (2025-12-28 05:59:32 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001018133s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
18 out of 60 is failed (threshold=3) [.X.XX..XX.X...X.X.XX....X...X.X..X.....X........X...X..X....] duration:
5.862554958s trace:
https://jaeger.dev.storj.io/trace/2df8490696898dfd
FAIL (2025-12-28 05:59:19 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 28 05:55:33.093317793 (542h26m32.964400828s) [upload=475685]: Starting upload
Dec 28 05:55:33.093367507 (542h26m32.964450542s) [upload=475685]: Done starting upload
Dec 28 05:55:33.093536659 (542h26m32.964619694s) [upload=475685]: Uploading segments...
Dec 28 05:55:33.093569472 (542h26m32.964652507s) [upload=475685]: Got next segment. Inline: false
Dec 28 05:55:33.093593899 (542h26m32.964676934s) [upload=475685]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 05:55:33.093597686 (542h26m32.964680721s) [upload=475685]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 05:55:33.339343235 (542h26m33.21042627s) [seg_pos={0 0}, upload=475685]: Begin upload segment...
Dec 28 05:55:33.339391968 (542h26m33.210475003s) [seg_pos={0 0}, upload=475685]: Done begin upload segment.
Dec 28 05:55:33.339427986 (542h26m33.210511031s) [noise=true , address=145.79.227.4:4187 , piece_id=RFGQBJKERSURKBDIWYC7JBME2N4DITZG3KJLJTEVJF2SZUAWHLWQ, seg_pos={0 0}, upload=475685]: Uploading piece...
Dec 28 05:55:33.340124412 (542h26m33.211207447s) [noise=true , address=145.79.227.18:4006 , piece_id=2BLQ5ARHTPFXAUXKO7DNPK7QZZXOOYGJASHWZTXD4MGY5PFUEUZA, seg_pos={0 0}, upload=475685]: Uploading piece...
Dec 28 05:55:33.340447347 (542h26m33.211530382s) [noise=true , address=145.79.227.14:4838 , piece_id=GRUDK6C5BUQGJQH7IPC7WFXSRRMTOJJOWOHPHPKVUOK7L5NV3EMA, seg_pos={0 0}, upload=475685]: Uploading piece...
Dec 28 05:55:33.340838633 (542h26m33.211921668s) [noise=true , address=77.90.31.11:4553 , piece_id=SMZ5XT7WJD7UIVLG4RXQOG5YAAYKH4A2UAK6FZ3YKY72GLQEP3JA, seg_pos={0 0}, upload=475685]: Uploading piece...
Dec 28 05:55:33.341125900 (542h26m33.212208935s) [noise=true , address=145.79.227.51:4410 , piece_id=LECHFPGJVNJZ42CQSEQ4T24NXHE66P5DECSXXWEKW6GD4AY6S7IQ, seg_pos={0 0}, upload=475685]: Uploading piece...
Dec 28 05:55:33.341419378 (542h26m33.212502413s) [noise=true , address=77.90.31.19:4841 , piece_id=W7EOGATNALKO2EOQIDV5EUVXLQ2XE6I4SKB5K3EXQFVQGIS4OZMQ, seg_pos={0 0}, upload=475685]: Uploading piece...
Dec 28 05:55:33.341732895 (542h26m33.21281593s) [noise=true , address=145.79.227.19:4079 , piece_id=6YTL65HQ7C66WN74HIJZ45D6ZZUJVNWZLWD3CRVKOBEZST4O5JMA, seg_pos={0 0}, upload=475685]: Uploading piece...
Dec 28 05:55:33.342080246 (542h26m33.213163292s) [noise=true , address=145.79.227.3:4150 , piece_id=XSN2WUKFGJPAZV7IKKFTVBCE3777W7QJPFINOZSI5DTSHZY2QT2Q, seg_pos={0 0}, upload=475685]: Uploading piece...
Dec 28 05:55:33.342590188 (542h26m33.213673233s) [noise=true , address=145.79.227.31:4798 , piece_id=TBLHTLG5V5JT5Y737Z77INRR325CH2LNGWZ6YMGCD335H3UHSKRA, seg_pos={0 0}, upload=475685]: Uploading piece...
Dec 28 05:55:33.343775314 (542h26m33.21485835s) [noise=true , address=145.79.227.17:4971 , piece_id=7QOHUJVVENFWRXANIKN6MWU4BUCC7ZSAIL4RAF5D4M3UR6Y3HKIA, seg_pos={0 0}, upload=475685]: Uploading piece...
Dec 28 05:55:33.344875540 (542h26m33.215958575s) [noise=true , address=145.79.227.49:4338 , piece_id=2WU4N4PYGZSZXU4QLZFG6EU7ADMND22XWSI7FECE4VMWGXLXU37Q, seg_pos={0 0}, upload=475685]: Uploading piece...
Dec 28 05:55:33.346640229 (542h26m33.217723265s) [noise=true , address=145.79.227.38:4979 , piece_id=YLHJRB6G2WPOU7RHFGUOON3S3KWXTNIO7BV3T3H7QY6GVHG63EXQ, seg_pos={0 0}, upload=475685]: Uploading piece...
Dec 28 05:55:33.347952999 (542h26m33.219036035s) [noise=true , address=45.91.116.7:4012 , piece_id=EUHICAKRSLDYW4HYTXOZTE6OEU4LIEQSWR6NPLPGE4EDZVRPYTGA, seg_pos={0 0}, upload=475685]: Uploading piece...
Dec 28 05:55:33.349149938 (542h26m33.220232973s) [noise=true , address=145.79.227.36:4518 , piece_id=MR22RW6NHOW5UDT6FGZZV6VNZWTUYYLT7QB2GHCWOOOPKHNQ7IQQ, seg_pos={0 0}, upload=475685]: Uploading piece...
Dec 28 05:55:33.349408171 (542h26m33.220491206s) [noise=true , address=145.79.227.48:4302 , piece_id=2JVUPRBXMI4NZAL6HAKPZUUHCBRJIGBJH4M5WGWNXGVCWTH34XDA, seg_pos={0 0}, upload=475685]: Uploading piece...
Dec 28 05:55:33.350929607 (542h26m33.222012662s) [noise=true , address=145.79.227.2:4115 , piece_id=QTSI2JB5QPYDE3S4HWTAPGYLYACLT35STUFF2XBX3JAK4XZE44TA, seg_pos={0 0}, upload=475685]: Uploading piece...
Dec 28 05:55:35.062950561 (542h26m34.934033596s) [upload=475685]: Next returned nil segment
Dec 28 05:55:35.062967303 (542h26m34.934050338s) [upload=475685]: Waiting for error group managing segments...
Dec 28 05:55:35.347184408 (542h26m35.218267443s) [noise=true , address=145.79.227.48:4302 , piece_id=2JVUPRBXMI4NZAL6HAKPZUUHCBRJIGBJH4M5WGWNXGVCWTH34XDA, seg_pos={0 0}, upload=475685]: Done uploading piece. err: <nil>
Dec 28 05:55:35.356826297 (542h26m35.227909332s) [noise=true , address=145.79.227.17:4971 , piece_id=7QOHUJVVENFWRXANIKN6MWU4BUCC7ZSAIL4RAF5D4M3UR6Y3HKIA, seg_pos={0 0}, upload=475685]: Done uploading piece. err: <nil>
Dec 28 05:55:35.360595244 (542h26m35.231678289s) [noise=true , address=145.79.227.38:4979 , piece_id=YLHJRB6G2WPOU7RHFGUOON3S3KWXTNIO7BV3T3H7QY6GVHG63EXQ, seg_pos={0 0}, upload=475685]: Done uploading piece. err: <nil>
Dec 28 05:55:35.361069647 (542h26m35.232152682s) [noise=true , address=145.79.227.3:4150 , piece_id=XSN2WUKFGJPAZV7IKKFTVBCE3777W7QJPFINOZSI5DTSHZY2QT2Q, seg_pos={0 0}, upload=475685]: Done uploading piece. err: <nil>
Dec 28 05:55:35.364315959 (542h26m35.235399004s) [noise=true , address=145.79.227.14:4838 , piece_id=GRUDK6C5BUQGJQH7IPC7WFXSRRMTOJJOWOHPHPKVUOK7L5NV3EMA, seg_pos={0 0}, upload=475685]: Done uploading piece. err: <nil>
Dec 28 05:55:35.372200572 (542h26m35.243283607s) [noise=true , address=145.79.227.36:4518 , piece_id=MR22RW6NHOW5UDT6FGZZV6VNZWTUYYLT7QB2GHCWOOOPKHNQ7IQQ, seg_pos={0 0}, upload=475685]: Done uploading piece. err: <nil>
Dec 28 05:55:35.377136982 (542h26m35.248220017s) [noise=true , address=145.79.227.51:4410 , piece_id=LECHFPGJVNJZ42CQSEQ4T24NXHE66P5DECSXXWEKW6GD4AY6S7IQ, seg_pos={0 0}, upload=475685]: Done uploading piece. err: <nil>
Dec 28 05:55:35.377586929 (542h26m35.248669994s) [noise=true , address=145.79.227.4:4187 , piece_id=RFGQBJKERSURKBDIWYC7JBME2N4DITZG3KJLJTEVJF2SZUAWHLWQ, seg_pos={0 0}, upload=475685]: Done uploading piece. err: <nil>
Dec 28 05:55:35.378320015 (542h26m35.24940305s) [noise=true , address=145.79.227.31:4798 , piece_id=TBLHTLG5V5JT5Y737Z77INRR325CH2LNGWZ6YMGCD335H3UHSKRA, seg_pos={0 0}, upload=475685]: Done uploading piece. err: <nil>
Dec 28 05:55:35.744060824 (542h26m35.615143869s) [noise=true , address=45.91.116.7:4012 , piece_id=EUHICAKRSLDYW4HYTXOZTE6OEU4LIEQSWR6NPLPGE4EDZVRPYTGA, seg_pos={0 0}, upload=475685]: Done uploading piece. err: <nil>
Dec 28 05:55:36.337187412 (542h26m36.208270447s) [noise=true , address=77.90.31.11:4553 , piece_id=SMZ5XT7WJD7UIVLG4RXQOG5YAAYKH4A2UAK6FZ3YKY72GLQEP3JA, seg_pos={0 0}, upload=475685]: Done uploading piece. err: <nil>
Dec 28 05:55:37.212533832 (542h26m37.083616877s) [noise=true , address=77.90.31.19:4841 , piece_id=W7EOGATNALKO2EOQIDV5EUVXLQ2XE6I4SKB5K3EXQFVQGIS4OZMQ, seg_pos={0 0}, upload=475685]: Done uploading piece. err: <nil>
Dec 28 05:56:02.402958638 (542h27m2.274041683s) [noise=true , address=145.79.227.19:4079 , piece_id=6YTL65HQ7C66WN74HIJZ45D6ZZUJVNWZLWD3CRVKOBEZST4O5JMA, seg_pos={0 0}, upload=475685]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:35836->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:35836->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35836->145.79.227.19:4079: use of closed network connection
Dec 28 05:56:02.402988726 (542h27m2.274071771s) [noise=true , address=145.79.227.18:4006 , piece_id=2BLQ5ARHTPFXAUXKO7DNPK7QZZXOOYGJASHWZTXD4MGY5PFUEUZA, seg_pos={0 0}, upload=475685]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: write tcp 188.166.225.170:34060->145.79.227.18:4006: use of closed network connection; write tcp 188.166.225.170:34060->145.79.227.18:4006: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34060->145.79.227.18:4006: use of closed network connection
Dec 28 05:56:02.403095910 (542h27m2.274178955s) [noise=true , address=145.79.227.2:4115 , piece_id=QTSI2JB5QPYDE3S4HWTAPGYLYACLT35STUFF2XBX3JAK4XZE44TA, seg_pos={0 0}, upload=475685]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: write tcp 188.166.225.170:37340->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:37340->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37340->145.79.227.2:4115: use of closed network connection
Dec 28 05:56:02.403104256 (542h27m2.274187302s) [noise=true , address=145.79.227.49:4338 , piece_id=2WU4N4PYGZSZXU4QLZFG6EU7ADMND22XWSI7FECE4VMWGXLXU37Q, seg_pos={0 0}, upload=475685]: Done uploading piece. err: ecclient: upload failed (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN, address:145.79.227.49:4338): protocol: write tcp 188.166.225.170:57176->145.79.227.49:4338: use of closed network connection; write tcp 188.166.225.170:57176->145.79.227.49:4338: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57176->145.79.227.49:4338: use of closed network connection
Dec 28 05:56:02.403136398 (542h27m2.274219443s) [seg_pos={0 0}, upload=475685]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 28 05:56:02.693153827 (542h27m2.564236872s) [upload=475685]: Done uploading segments...
Dec 28 05:56:02.693202830 (542h27m2.564285875s) [upload=475685]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.862589664s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
26 out of 60 is failed (threshold=6) [XXX....XXXX....XXX.X...X....X...XX.X..XXX....X..XX.X...X.X.X] duration:
5.003025897s trace:
https://jaeger.dev.storj.io/trace/6cc36b9248533147
FAIL (2025-12-28 05:59:03 UTC)
orbiter warning: context deadline exceeded
Duration: 5.003048799s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [...............X.........X...X....................X.........] duration:
27.213205781s trace:
https://jaeger.dev.storj.io/trace/35a88c4b3a3d78b6
FAIL (2025-12-28 05:59:01 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH, Piece ID: HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA) context deadline exceeded; piecestore: (Node ID: 1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, Piece ID: GY4LXIIIB7SVWA2WRAEHJVB6HLN6XQYISF7ZPVE4KA42C23MEQ4A) context deadline exceeded; piecestore: (Node ID: 1RQ3vtArvKAvaMJTVa9VDDVdWR5oBuZ7gaPB3pzc1MouqBZhWu, Piece ID: FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ) context deadline exceeded; piecestore: (Node ID: 127KQ2X6VXddEeFRMau7MVGwtTsUmFsm8hTYDR2A6B4A1oVKfyo, Piece ID: V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ) context deadline exceeded
======LOGS======
) [noise=true , address=145.79.227.44:4209 , piece_id=JRNDKXGHFY5OJYSY4ESURYMK5BNRDWY6INLHOWEK55KHMNRJO4AQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.655023976 (542h19m51.526107021s) [noise=true , address=77.90.31.13:4647 , piece_id=SOPU7ZUZQXRLU6TUL4L3ODX7U7WFAKUDOSK5UO66WMAIANU6PVUA, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.655676448 (542h19m51.526759493s) [noise=true , address=45.91.116.8:4059 , piece_id=OSA2YK4AH2DPQBNSJDFT3PLTGPJM6N7EOB7X27ZP2WM52HWSHT4Q, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.656204463 (542h19m51.527287509s) [noise=true , address=145.79.227.49:4370 , piece_id=QDMDBC3576X7ZYGUAC2DLJKWBNVOLSROCMA4UNFDZ47TTADILUAQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.656699927 (542h19m51.527782972s) [noise=true , address=45.91.116.9:4290 , piece_id=MP5DAA5UGHWQ3BA5R7GBWNIV2B4JYCL5WUSV2PY7GTO4PNLH2F2Q, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.657183808 (542h19m51.528266883s) [noise=true , address=77.90.31.18:4827 , piece_id=V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.657682177 (542h19m51.528765222s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.658706768 (542h19m51.529789813s) [noise=true , address=145.79.227.33:4876 , piece_id=E5QQDATADJON5NOOQ6X37GFKUQCFUKJRGK6G2EURBSUJVLF72Y7A, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.660113707 (542h19m51.531196752s) [noise=true , address=145.79.227.34:4915 , piece_id=MR5WOODFPXH57I6O2JCMC3EE5OWXF23CFLLATZXFP5XK6YPWDZJA, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.661356113 (542h19m51.532439158s) [noise=true , address=77.90.31.15:4727 , piece_id=3ACH3RUAR5KUXZU23AL3OQSGIZQOU5MFOFRN3JUYLQJBJCA4JOZQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.664431910 (542h19m51.535514955s) [noise=true , address=45.91.116.6:4998 , piece_id=FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.666377204 (542h19m51.537460249s) [noise=true , address=145.79.227.40:4067 , piece_id=36Q7DHHN5ELVD3BYSNUGJ7WYXNGIDKERTOPRAZJNJ7YQTOUPWEYQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.666754502 (542h19m51.537837548s) [noise=true , address=77.90.31.12:4594 , piece_id=DNXKJQOUHLAWMOCR35THP5CKCB6LZBW4GMJU6O6KKVSIJ3ALLONQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.668451284 (542h19m51.539534319s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.671076552 (542h19m51.542159587s) [noise=true , address=77.90.31.11:4560 , piece_id=HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.674568571 (542h19m51.545651626s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:52.998088476 (542h19m52.869171521s) [upload=475588]: Next returned nil segment
Dec 28 05:48:52.998103846 (542h19m52.869186881s) [upload=475588]: Waiting for error group managing segments...
Dec 28 05:48:53.543654582 (542h19m53.414737627s) [noise=true , address=145.79.227.40:4067 , piece_id=36Q7DHHN5ELVD3BYSNUGJ7WYXNGIDKERTOPRAZJNJ7YQTOUPWEYQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.543744133 (542h19m53.414827178s) [noise=true , address=145.79.227.49:4370 , piece_id=QDMDBC3576X7ZYGUAC2DLJKWBNVOLSROCMA4UNFDZ47TTADILUAQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.556254274 (542h19m53.427337309s) [noise=true , address=145.79.227.34:4915 , piece_id=MR5WOODFPXH57I6O2JCMC3EE5OWXF23CFLLATZXFP5XK6YPWDZJA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.810781113 (542h19m53.681864158s) [noise=true , address=77.90.31.13:4647 , piece_id=SOPU7ZUZQXRLU6TUL4L3ODX7U7WFAKUDOSK5UO66WMAIANU6PVUA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.814762255 (542h19m53.6858453s) [noise=true , address=77.90.31.16:4756 , piece_id=GY4LXIIIB7SVWA2WRAEHJVB6HLN6XQYISF7ZPVE4KA42C23MEQ4A, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.816966662 (542h19m53.688049707s) [noise=true , address=77.90.31.15:4727 , piece_id=3ACH3RUAR5KUXZU23AL3OQSGIZQOU5MFOFRN3JUYLQJBJCA4JOZQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.895077718 (542h19m53.766160783s) [noise=true , address=45.91.116.9:4290 , piece_id=MP5DAA5UGHWQ3BA5R7GBWNIV2B4JYCL5WUSV2PY7GTO4PNLH2F2Q, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.906586933 (542h19m53.777669978s) [noise=true , address=45.91.116.8:4059 , piece_id=OSA2YK4AH2DPQBNSJDFT3PLTGPJM6N7EOB7X27ZP2WM52HWSHT4Q, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.930785842 (542h19m53.801868877s) [noise=true , address=145.79.227.33:4876 , piece_id=E5QQDATADJON5NOOQ6X37GFKUQCFUKJRGK6G2EURBSUJVLF72Y7A, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.087446700 (542h19m53.958529735s) [noise=true , address=45.91.116.6:4998 , piece_id=FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.129981903 (542h19m54.001064938s) [noise=true , address=77.90.31.11:4560 , piece_id=HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.264740966 (542h19m54.135824011s) [noise=true , address=77.90.31.12:4594 , piece_id=DNXKJQOUHLAWMOCR35THP5CKCB6LZBW4GMJU6O6KKVSIJ3ALLONQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.368348401 (542h19m54.239431446s) [noise=true , address=77.90.31.18:4827 , piece_id=V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.437713885 (542h19m54.30879693s) [noise=true , address=145.79.227.44:4209 , piece_id=JRNDKXGHFY5OJYSY4ESURYMK5BNRDWY6INLHOWEK55KHMNRJO4AQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.437828132 (542h19m54.308911187s) [seg_pos={0 0}, upload=475588]: Segment reached optimal threshold of 14 pieces.
Dec 28 05:48:54.438064122 (542h19m54.309147167s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF
Dec 28 05:48:54.438074481 (542h19m54.309157546s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold
Dec 28 05:48:54.438099619 (542h19m54.309182664s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F4fJAGJM2ZPRt7vJH2cZmEbYeujmxcfi7aPZ2g9VPTNYBYRAi): protocol: expected piece hash; context canceled; EOF
Dec 28 05:48:54.438108586 (542h19m54.309191631s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold
Dec 28 05:48:54.438243403 (542h19m54.309326458s) [noise=true , address=77.90.31.19:4876 , piece_id=PUIQX7IZSNPKZTZTYH32KTFUM7K53KWZASLPT6JHAPMOB2JCXPGA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12H3UNBGeAufYSdVvEgYkv2QWeFtjKMvHvYGeqGeBvjEHqDDjBB): protocol: expected piece hash; context canceled; EOF
Dec 28 05:48:54.438252169 (542h19m54.309335214s) [noise=true , address=77.90.31.19:4876 , piece_id=PUIQX7IZSNPKZTZTYH32KTFUM7K53KWZASLPT6JHAPMOB2JCXPGA, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold
Dec 28 05:48:54.438391024 (542h19m54.309474069s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection
Dec 28 05:48:54.438398588 (542h19m54.309481633s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold
Dec 28 05:48:54.438699081 (542h19m54.309782126s) [seg_pos={0 0}, upload=475588]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 05:48:54.438794753 (542h19m54.309877798s) [upload=475588]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 05:48:54.438801255 (542h19m54.30988429s) [upload=475588]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 05:48:54.728435695 (542h19m54.59951876s) [upload=475588]: Done uploading segments...
Dec 28 05:48:54.728457938 (542h19m54.599540983s) [upload=475588]: Upload finished. err: <nil>
Duration: 27.213232382s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
19 out of 60 is failed (threshold=3) [X.X.XX..XX.X...X.X.XX....X...X.X..X.....X........X...X..X...] duration:
5.688634435s trace:
https://jaeger.dev.storj.io/trace/2df8490696898dfd
FAIL (2025-12-28 05:58:28 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 28 05:55:33.093317793 (542h26m32.964400828s) [upload=475685]: Starting upload
Dec 28 05:55:33.093367507 (542h26m32.964450542s) [upload=475685]: Done starting upload
Dec 28 05:55:33.093536659 (542h26m32.964619694s) [upload=475685]: Uploading segments...
Dec 28 05:55:33.093569472 (542h26m32.964652507s) [upload=475685]: Got next segment. Inline: false
Dec 28 05:55:33.093593899 (542h26m32.964676934s) [upload=475685]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 05:55:33.093597686 (542h26m32.964680721s) [upload=475685]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 05:55:33.339343235 (542h26m33.21042627s) [seg_pos={0 0}, upload=475685]: Begin upload segment...
Dec 28 05:55:33.339391968 (542h26m33.210475003s) [seg_pos={0 0}, upload=475685]: Done begin upload segment.
Dec 28 05:55:33.339427986 (542h26m33.210511031s) [noise=true , address=145.79.227.4:4187 , piece_id=RFGQBJKERSURKBDIWYC7JBME2N4DITZG3KJLJTEVJF2SZUAWHLWQ, seg_pos={0 0}, upload=475685]: Uploading piece...
Dec 28 05:55:33.340124412 (542h26m33.211207447s) [noise=true , address=145.79.227.18:4006 , piece_id=2BLQ5ARHTPFXAUXKO7DNPK7QZZXOOYGJASHWZTXD4MGY5PFUEUZA, seg_pos={0 0}, upload=475685]: Uploading piece...
Dec 28 05:55:33.340447347 (542h26m33.211530382s) [noise=true , address=145.79.227.14:4838 , piece_id=GRUDK6C5BUQGJQH7IPC7WFXSRRMTOJJOWOHPHPKVUOK7L5NV3EMA, seg_pos={0 0}, upload=475685]: Uploading piece...
Dec 28 05:55:33.340838633 (542h26m33.211921668s) [noise=true , address=77.90.31.11:4553 , piece_id=SMZ5XT7WJD7UIVLG4RXQOG5YAAYKH4A2UAK6FZ3YKY72GLQEP3JA, seg_pos={0 0}, upload=475685]: Uploading piece...
Dec 28 05:55:33.341125900 (542h26m33.212208935s) [noise=true , address=145.79.227.51:4410 , piece_id=LECHFPGJVNJZ42CQSEQ4T24NXHE66P5DECSXXWEKW6GD4AY6S7IQ, seg_pos={0 0}, upload=475685]: Uploading piece...
Dec 28 05:55:33.341419378 (542h26m33.212502413s) [noise=true , address=77.90.31.19:4841 , piece_id=W7EOGATNALKO2EOQIDV5EUVXLQ2XE6I4SKB5K3EXQFVQGIS4OZMQ, seg_pos={0 0}, upload=475685]: Uploading piece...
Dec 28 05:55:33.341732895 (542h26m33.21281593s) [noise=true , address=145.79.227.19:4079 , piece_id=6YTL65HQ7C66WN74HIJZ45D6ZZUJVNWZLWD3CRVKOBEZST4O5JMA, seg_pos={0 0}, upload=475685]: Uploading piece...
Dec 28 05:55:33.342080246 (542h26m33.213163292s) [noise=true , address=145.79.227.3:4150 , piece_id=XSN2WUKFGJPAZV7IKKFTVBCE3777W7QJPFINOZSI5DTSHZY2QT2Q, seg_pos={0 0}, upload=475685]: Uploading piece...
Dec 28 05:55:33.342590188 (542h26m33.213673233s) [noise=true , address=145.79.227.31:4798 , piece_id=TBLHTLG5V5JT5Y737Z77INRR325CH2LNGWZ6YMGCD335H3UHSKRA, seg_pos={0 0}, upload=475685]: Uploading piece...
Dec 28 05:55:33.343775314 (542h26m33.21485835s) [noise=true , address=145.79.227.17:4971 , piece_id=7QOHUJVVENFWRXANIKN6MWU4BUCC7ZSAIL4RAF5D4M3UR6Y3HKIA, seg_pos={0 0}, upload=475685]: Uploading piece...
Dec 28 05:55:33.344875540 (542h26m33.215958575s) [noise=true , address=145.79.227.49:4338 , piece_id=2WU4N4PYGZSZXU4QLZFG6EU7ADMND22XWSI7FECE4VMWGXLXU37Q, seg_pos={0 0}, upload=475685]: Uploading piece...
Dec 28 05:55:33.346640229 (542h26m33.217723265s) [noise=true , address=145.79.227.38:4979 , piece_id=YLHJRB6G2WPOU7RHFGUOON3S3KWXTNIO7BV3T3H7QY6GVHG63EXQ, seg_pos={0 0}, upload=475685]: Uploading piece...
Dec 28 05:55:33.347952999 (542h26m33.219036035s) [noise=true , address=45.91.116.7:4012 , piece_id=EUHICAKRSLDYW4HYTXOZTE6OEU4LIEQSWR6NPLPGE4EDZVRPYTGA, seg_pos={0 0}, upload=475685]: Uploading piece...
Dec 28 05:55:33.349149938 (542h26m33.220232973s) [noise=true , address=145.79.227.36:4518 , piece_id=MR22RW6NHOW5UDT6FGZZV6VNZWTUYYLT7QB2GHCWOOOPKHNQ7IQQ, seg_pos={0 0}, upload=475685]: Uploading piece...
Dec 28 05:55:33.349408171 (542h26m33.220491206s) [noise=true , address=145.79.227.48:4302 , piece_id=2JVUPRBXMI4NZAL6HAKPZUUHCBRJIGBJH4M5WGWNXGVCWTH34XDA, seg_pos={0 0}, upload=475685]: Uploading piece...
Dec 28 05:55:33.350929607 (542h26m33.222012662s) [noise=true , address=145.79.227.2:4115 , piece_id=QTSI2JB5QPYDE3S4HWTAPGYLYACLT35STUFF2XBX3JAK4XZE44TA, seg_pos={0 0}, upload=475685]: Uploading piece...
Dec 28 05:55:35.062950561 (542h26m34.934033596s) [upload=475685]: Next returned nil segment
Dec 28 05:55:35.062967303 (542h26m34.934050338s) [upload=475685]: Waiting for error group managing segments...
Dec 28 05:55:35.347184408 (542h26m35.218267443s) [noise=true , address=145.79.227.48:4302 , piece_id=2JVUPRBXMI4NZAL6HAKPZUUHCBRJIGBJH4M5WGWNXGVCWTH34XDA, seg_pos={0 0}, upload=475685]: Done uploading piece. err: <nil>
Dec 28 05:55:35.356826297 (542h26m35.227909332s) [noise=true , address=145.79.227.17:4971 , piece_id=7QOHUJVVENFWRXANIKN6MWU4BUCC7ZSAIL4RAF5D4M3UR6Y3HKIA, seg_pos={0 0}, upload=475685]: Done uploading piece. err: <nil>
Dec 28 05:55:35.360595244 (542h26m35.231678289s) [noise=true , address=145.79.227.38:4979 , piece_id=YLHJRB6G2WPOU7RHFGUOON3S3KWXTNIO7BV3T3H7QY6GVHG63EXQ, seg_pos={0 0}, upload=475685]: Done uploading piece. err: <nil>
Dec 28 05:55:35.361069647 (542h26m35.232152682s) [noise=true , address=145.79.227.3:4150 , piece_id=XSN2WUKFGJPAZV7IKKFTVBCE3777W7QJPFINOZSI5DTSHZY2QT2Q, seg_pos={0 0}, upload=475685]: Done uploading piece. err: <nil>
Dec 28 05:55:35.364315959 (542h26m35.235399004s) [noise=true , address=145.79.227.14:4838 , piece_id=GRUDK6C5BUQGJQH7IPC7WFXSRRMTOJJOWOHPHPKVUOK7L5NV3EMA, seg_pos={0 0}, upload=475685]: Done uploading piece. err: <nil>
Dec 28 05:55:35.372200572 (542h26m35.243283607s) [noise=true , address=145.79.227.36:4518 , piece_id=MR22RW6NHOW5UDT6FGZZV6VNZWTUYYLT7QB2GHCWOOOPKHNQ7IQQ, seg_pos={0 0}, upload=475685]: Done uploading piece. err: <nil>
Dec 28 05:55:35.377136982 (542h26m35.248220017s) [noise=true , address=145.79.227.51:4410 , piece_id=LECHFPGJVNJZ42CQSEQ4T24NXHE66P5DECSXXWEKW6GD4AY6S7IQ, seg_pos={0 0}, upload=475685]: Done uploading piece. err: <nil>
Dec 28 05:55:35.377586929 (542h26m35.248669994s) [noise=true , address=145.79.227.4:4187 , piece_id=RFGQBJKERSURKBDIWYC7JBME2N4DITZG3KJLJTEVJF2SZUAWHLWQ, seg_pos={0 0}, upload=475685]: Done uploading piece. err: <nil>
Dec 28 05:55:35.378320015 (542h26m35.24940305s) [noise=true , address=145.79.227.31:4798 , piece_id=TBLHTLG5V5JT5Y737Z77INRR325CH2LNGWZ6YMGCD335H3UHSKRA, seg_pos={0 0}, upload=475685]: Done uploading piece. err: <nil>
Dec 28 05:55:35.744060824 (542h26m35.615143869s) [noise=true , address=45.91.116.7:4012 , piece_id=EUHICAKRSLDYW4HYTXOZTE6OEU4LIEQSWR6NPLPGE4EDZVRPYTGA, seg_pos={0 0}, upload=475685]: Done uploading piece. err: <nil>
Dec 28 05:55:36.337187412 (542h26m36.208270447s) [noise=true , address=77.90.31.11:4553 , piece_id=SMZ5XT7WJD7UIVLG4RXQOG5YAAYKH4A2UAK6FZ3YKY72GLQEP3JA, seg_pos={0 0}, upload=475685]: Done uploading piece. err: <nil>
Dec 28 05:55:37.212533832 (542h26m37.083616877s) [noise=true , address=77.90.31.19:4841 , piece_id=W7EOGATNALKO2EOQIDV5EUVXLQ2XE6I4SKB5K3EXQFVQGIS4OZMQ, seg_pos={0 0}, upload=475685]: Done uploading piece. err: <nil>
Dec 28 05:56:02.402958638 (542h27m2.274041683s) [noise=true , address=145.79.227.19:4079 , piece_id=6YTL65HQ7C66WN74HIJZ45D6ZZUJVNWZLWD3CRVKOBEZST4O5JMA, seg_pos={0 0}, upload=475685]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:35836->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:35836->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35836->145.79.227.19:4079: use of closed network connection
Dec 28 05:56:02.402988726 (542h27m2.274071771s) [noise=true , address=145.79.227.18:4006 , piece_id=2BLQ5ARHTPFXAUXKO7DNPK7QZZXOOYGJASHWZTXD4MGY5PFUEUZA, seg_pos={0 0}, upload=475685]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: write tcp 188.166.225.170:34060->145.79.227.18:4006: use of closed network connection; write tcp 188.166.225.170:34060->145.79.227.18:4006: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34060->145.79.227.18:4006: use of closed network connection
Dec 28 05:56:02.403095910 (542h27m2.274178955s) [noise=true , address=145.79.227.2:4115 , piece_id=QTSI2JB5QPYDE3S4HWTAPGYLYACLT35STUFF2XBX3JAK4XZE44TA, seg_pos={0 0}, upload=475685]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: write tcp 188.166.225.170:37340->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:37340->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37340->145.79.227.2:4115: use of closed network connection
Dec 28 05:56:02.403104256 (542h27m2.274187302s) [noise=true , address=145.79.227.49:4338 , piece_id=2WU4N4PYGZSZXU4QLZFG6EU7ADMND22XWSI7FECE4VMWGXLXU37Q, seg_pos={0 0}, upload=475685]: Done uploading piece. err: ecclient: upload failed (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN, address:145.79.227.49:4338): protocol: write tcp 188.166.225.170:57176->145.79.227.49:4338: use of closed network connection; write tcp 188.166.225.170:57176->145.79.227.49:4338: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57176->145.79.227.49:4338: use of closed network connection
Dec 28 05:56:02.403136398 (542h27m2.274219443s) [seg_pos={0 0}, upload=475685]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 28 05:56:02.693153827 (542h27m2.564236872s) [upload=475685]: Done uploading segments...
Dec 28 05:56:02.693202830 (542h27m2.564285875s) [upload=475685]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.688662839s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [................X.........X...X....................X........] duration:
38.758818605s trace:
https://jaeger.dev.storj.io/trace/35a88c4b3a3d78b6
FAIL (2025-12-28 05:58:01 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH, Piece ID: HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA) context deadline exceeded; piecestore: (Node ID: 1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, Piece ID: GY4LXIIIB7SVWA2WRAEHJVB6HLN6XQYISF7ZPVE4KA42C23MEQ4A) context deadline exceeded; piecestore: (Node ID: 1RQ3vtArvKAvaMJTVa9VDDVdWR5oBuZ7gaPB3pzc1MouqBZhWu, Piece ID: FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ) context deadline exceeded; piecestore: (Node ID: 127KQ2X6VXddEeFRMau7MVGwtTsUmFsm8hTYDR2A6B4A1oVKfyo, Piece ID: V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ) context deadline exceeded
======LOGS======
) [noise=true , address=145.79.227.44:4209 , piece_id=JRNDKXGHFY5OJYSY4ESURYMK5BNRDWY6INLHOWEK55KHMNRJO4AQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.655023976 (542h19m51.526107021s) [noise=true , address=77.90.31.13:4647 , piece_id=SOPU7ZUZQXRLU6TUL4L3ODX7U7WFAKUDOSK5UO66WMAIANU6PVUA, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.655676448 (542h19m51.526759493s) [noise=true , address=45.91.116.8:4059 , piece_id=OSA2YK4AH2DPQBNSJDFT3PLTGPJM6N7EOB7X27ZP2WM52HWSHT4Q, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.656204463 (542h19m51.527287509s) [noise=true , address=145.79.227.49:4370 , piece_id=QDMDBC3576X7ZYGUAC2DLJKWBNVOLSROCMA4UNFDZ47TTADILUAQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.656699927 (542h19m51.527782972s) [noise=true , address=45.91.116.9:4290 , piece_id=MP5DAA5UGHWQ3BA5R7GBWNIV2B4JYCL5WUSV2PY7GTO4PNLH2F2Q, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.657183808 (542h19m51.528266883s) [noise=true , address=77.90.31.18:4827 , piece_id=V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.657682177 (542h19m51.528765222s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.658706768 (542h19m51.529789813s) [noise=true , address=145.79.227.33:4876 , piece_id=E5QQDATADJON5NOOQ6X37GFKUQCFUKJRGK6G2EURBSUJVLF72Y7A, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.660113707 (542h19m51.531196752s) [noise=true , address=145.79.227.34:4915 , piece_id=MR5WOODFPXH57I6O2JCMC3EE5OWXF23CFLLATZXFP5XK6YPWDZJA, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.661356113 (542h19m51.532439158s) [noise=true , address=77.90.31.15:4727 , piece_id=3ACH3RUAR5KUXZU23AL3OQSGIZQOU5MFOFRN3JUYLQJBJCA4JOZQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.664431910 (542h19m51.535514955s) [noise=true , address=45.91.116.6:4998 , piece_id=FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.666377204 (542h19m51.537460249s) [noise=true , address=145.79.227.40:4067 , piece_id=36Q7DHHN5ELVD3BYSNUGJ7WYXNGIDKERTOPRAZJNJ7YQTOUPWEYQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.666754502 (542h19m51.537837548s) [noise=true , address=77.90.31.12:4594 , piece_id=DNXKJQOUHLAWMOCR35THP5CKCB6LZBW4GMJU6O6KKVSIJ3ALLONQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.668451284 (542h19m51.539534319s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.671076552 (542h19m51.542159587s) [noise=true , address=77.90.31.11:4560 , piece_id=HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.674568571 (542h19m51.545651626s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:52.998088476 (542h19m52.869171521s) [upload=475588]: Next returned nil segment
Dec 28 05:48:52.998103846 (542h19m52.869186881s) [upload=475588]: Waiting for error group managing segments...
Dec 28 05:48:53.543654582 (542h19m53.414737627s) [noise=true , address=145.79.227.40:4067 , piece_id=36Q7DHHN5ELVD3BYSNUGJ7WYXNGIDKERTOPRAZJNJ7YQTOUPWEYQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.543744133 (542h19m53.414827178s) [noise=true , address=145.79.227.49:4370 , piece_id=QDMDBC3576X7ZYGUAC2DLJKWBNVOLSROCMA4UNFDZ47TTADILUAQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.556254274 (542h19m53.427337309s) [noise=true , address=145.79.227.34:4915 , piece_id=MR5WOODFPXH57I6O2JCMC3EE5OWXF23CFLLATZXFP5XK6YPWDZJA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.810781113 (542h19m53.681864158s) [noise=true , address=77.90.31.13:4647 , piece_id=SOPU7ZUZQXRLU6TUL4L3ODX7U7WFAKUDOSK5UO66WMAIANU6PVUA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.814762255 (542h19m53.6858453s) [noise=true , address=77.90.31.16:4756 , piece_id=GY4LXIIIB7SVWA2WRAEHJVB6HLN6XQYISF7ZPVE4KA42C23MEQ4A, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.816966662 (542h19m53.688049707s) [noise=true , address=77.90.31.15:4727 , piece_id=3ACH3RUAR5KUXZU23AL3OQSGIZQOU5MFOFRN3JUYLQJBJCA4JOZQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.895077718 (542h19m53.766160783s) [noise=true , address=45.91.116.9:4290 , piece_id=MP5DAA5UGHWQ3BA5R7GBWNIV2B4JYCL5WUSV2PY7GTO4PNLH2F2Q, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.906586933 (542h19m53.777669978s) [noise=true , address=45.91.116.8:4059 , piece_id=OSA2YK4AH2DPQBNSJDFT3PLTGPJM6N7EOB7X27ZP2WM52HWSHT4Q, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.930785842 (542h19m53.801868877s) [noise=true , address=145.79.227.33:4876 , piece_id=E5QQDATADJON5NOOQ6X37GFKUQCFUKJRGK6G2EURBSUJVLF72Y7A, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.087446700 (542h19m53.958529735s) [noise=true , address=45.91.116.6:4998 , piece_id=FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.129981903 (542h19m54.001064938s) [noise=true , address=77.90.31.11:4560 , piece_id=HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.264740966 (542h19m54.135824011s) [noise=true , address=77.90.31.12:4594 , piece_id=DNXKJQOUHLAWMOCR35THP5CKCB6LZBW4GMJU6O6KKVSIJ3ALLONQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.368348401 (542h19m54.239431446s) [noise=true , address=77.90.31.18:4827 , piece_id=V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.437713885 (542h19m54.30879693s) [noise=true , address=145.79.227.44:4209 , piece_id=JRNDKXGHFY5OJYSY4ESURYMK5BNRDWY6INLHOWEK55KHMNRJO4AQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.437828132 (542h19m54.308911187s) [seg_pos={0 0}, upload=475588]: Segment reached optimal threshold of 14 pieces.
Dec 28 05:48:54.438064122 (542h19m54.309147167s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF
Dec 28 05:48:54.438074481 (542h19m54.309157546s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold
Dec 28 05:48:54.438099619 (542h19m54.309182664s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F4fJAGJM2ZPRt7vJH2cZmEbYeujmxcfi7aPZ2g9VPTNYBYRAi): protocol: expected piece hash; context canceled; EOF
Dec 28 05:48:54.438108586 (542h19m54.309191631s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold
Dec 28 05:48:54.438243403 (542h19m54.309326458s) [noise=true , address=77.90.31.19:4876 , piece_id=PUIQX7IZSNPKZTZTYH32KTFUM7K53KWZASLPT6JHAPMOB2JCXPGA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12H3UNBGeAufYSdVvEgYkv2QWeFtjKMvHvYGeqGeBvjEHqDDjBB): protocol: expected piece hash; context canceled; EOF
Dec 28 05:48:54.438252169 (542h19m54.309335214s) [noise=true , address=77.90.31.19:4876 , piece_id=PUIQX7IZSNPKZTZTYH32KTFUM7K53KWZASLPT6JHAPMOB2JCXPGA, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold
Dec 28 05:48:54.438391024 (542h19m54.309474069s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection
Dec 28 05:48:54.438398588 (542h19m54.309481633s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold
Dec 28 05:48:54.438699081 (542h19m54.309782126s) [seg_pos={0 0}, upload=475588]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 05:48:54.438794753 (542h19m54.309877798s) [upload=475588]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 05:48:54.438801255 (542h19m54.30988429s) [upload=475588]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 05:48:54.728435695 (542h19m54.59951876s) [upload=475588]: Done uploading segments...
Dec 28 05:48:54.728457938 (542h19m54.599540983s) [upload=475588]: Upload finished. err: <nil>
Duration: 38.758845887s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
25 out of 60 is failed (threshold=6) [.XXX....XXXX....XXX.X...X....X...XX.X..XXX....X..XX.X...X.X.] duration:
3.488320343s trace:
https://jaeger.dev.storj.io/trace/444fe9e1aa1340ed
FAIL (2025-12-28 05:57:50 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.48835533s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
20 out of 60 is failed (threshold=3) [XX.X.XX..XX.X...X.X.XX....X...X.X..X.....X........X...X..X..] duration:
5.989203805s trace:
https://jaeger.dev.storj.io/trace/2df8490696898dfd
FAIL (2025-12-28 05:57:45 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 28 05:55:33.093317793 (542h26m32.964400828s) [upload=475685]: Starting upload
Dec 28 05:55:33.093367507 (542h26m32.964450542s) [upload=475685]: Done starting upload
Dec 28 05:55:33.093536659 (542h26m32.964619694s) [upload=475685]: Uploading segments...
Dec 28 05:55:33.093569472 (542h26m32.964652507s) [upload=475685]: Got next segment. Inline: false
Dec 28 05:55:33.093593899 (542h26m32.964676934s) [upload=475685]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 05:55:33.093597686 (542h26m32.964680721s) [upload=475685]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 05:55:33.339343235 (542h26m33.21042627s) [seg_pos={0 0}, upload=475685]: Begin upload segment...
Dec 28 05:55:33.339391968 (542h26m33.210475003s) [seg_pos={0 0}, upload=475685]: Done begin upload segment.
Dec 28 05:55:33.339427986 (542h26m33.210511031s) [noise=true , address=145.79.227.4:4187 , piece_id=RFGQBJKERSURKBDIWYC7JBME2N4DITZG3KJLJTEVJF2SZUAWHLWQ, seg_pos={0 0}, upload=475685]: Uploading piece...
Dec 28 05:55:33.340124412 (542h26m33.211207447s) [noise=true , address=145.79.227.18:4006 , piece_id=2BLQ5ARHTPFXAUXKO7DNPK7QZZXOOYGJASHWZTXD4MGY5PFUEUZA, seg_pos={0 0}, upload=475685]: Uploading piece...
Dec 28 05:55:33.340447347 (542h26m33.211530382s) [noise=true , address=145.79.227.14:4838 , piece_id=GRUDK6C5BUQGJQH7IPC7WFXSRRMTOJJOWOHPHPKVUOK7L5NV3EMA, seg_pos={0 0}, upload=475685]: Uploading piece...
Dec 28 05:55:33.340838633 (542h26m33.211921668s) [noise=true , address=77.90.31.11:4553 , piece_id=SMZ5XT7WJD7UIVLG4RXQOG5YAAYKH4A2UAK6FZ3YKY72GLQEP3JA, seg_pos={0 0}, upload=475685]: Uploading piece...
Dec 28 05:55:33.341125900 (542h26m33.212208935s) [noise=true , address=145.79.227.51:4410 , piece_id=LECHFPGJVNJZ42CQSEQ4T24NXHE66P5DECSXXWEKW6GD4AY6S7IQ, seg_pos={0 0}, upload=475685]: Uploading piece...
Dec 28 05:55:33.341419378 (542h26m33.212502413s) [noise=true , address=77.90.31.19:4841 , piece_id=W7EOGATNALKO2EOQIDV5EUVXLQ2XE6I4SKB5K3EXQFVQGIS4OZMQ, seg_pos={0 0}, upload=475685]: Uploading piece...
Dec 28 05:55:33.341732895 (542h26m33.21281593s) [noise=true , address=145.79.227.19:4079 , piece_id=6YTL65HQ7C66WN74HIJZ45D6ZZUJVNWZLWD3CRVKOBEZST4O5JMA, seg_pos={0 0}, upload=475685]: Uploading piece...
Dec 28 05:55:33.342080246 (542h26m33.213163292s) [noise=true , address=145.79.227.3:4150 , piece_id=XSN2WUKFGJPAZV7IKKFTVBCE3777W7QJPFINOZSI5DTSHZY2QT2Q, seg_pos={0 0}, upload=475685]: Uploading piece...
Dec 28 05:55:33.342590188 (542h26m33.213673233s) [noise=true , address=145.79.227.31:4798 , piece_id=TBLHTLG5V5JT5Y737Z77INRR325CH2LNGWZ6YMGCD335H3UHSKRA, seg_pos={0 0}, upload=475685]: Uploading piece...
Dec 28 05:55:33.343775314 (542h26m33.21485835s) [noise=true , address=145.79.227.17:4971 , piece_id=7QOHUJVVENFWRXANIKN6MWU4BUCC7ZSAIL4RAF5D4M3UR6Y3HKIA, seg_pos={0 0}, upload=475685]: Uploading piece...
Dec 28 05:55:33.344875540 (542h26m33.215958575s) [noise=true , address=145.79.227.49:4338 , piece_id=2WU4N4PYGZSZXU4QLZFG6EU7ADMND22XWSI7FECE4VMWGXLXU37Q, seg_pos={0 0}, upload=475685]: Uploading piece...
Dec 28 05:55:33.346640229 (542h26m33.217723265s) [noise=true , address=145.79.227.38:4979 , piece_id=YLHJRB6G2WPOU7RHFGUOON3S3KWXTNIO7BV3T3H7QY6GVHG63EXQ, seg_pos={0 0}, upload=475685]: Uploading piece...
Dec 28 05:55:33.347952999 (542h26m33.219036035s) [noise=true , address=45.91.116.7:4012 , piece_id=EUHICAKRSLDYW4HYTXOZTE6OEU4LIEQSWR6NPLPGE4EDZVRPYTGA, seg_pos={0 0}, upload=475685]: Uploading piece...
Dec 28 05:55:33.349149938 (542h26m33.220232973s) [noise=true , address=145.79.227.36:4518 , piece_id=MR22RW6NHOW5UDT6FGZZV6VNZWTUYYLT7QB2GHCWOOOPKHNQ7IQQ, seg_pos={0 0}, upload=475685]: Uploading piece...
Dec 28 05:55:33.349408171 (542h26m33.220491206s) [noise=true , address=145.79.227.48:4302 , piece_id=2JVUPRBXMI4NZAL6HAKPZUUHCBRJIGBJH4M5WGWNXGVCWTH34XDA, seg_pos={0 0}, upload=475685]: Uploading piece...
Dec 28 05:55:33.350929607 (542h26m33.222012662s) [noise=true , address=145.79.227.2:4115 , piece_id=QTSI2JB5QPYDE3S4HWTAPGYLYACLT35STUFF2XBX3JAK4XZE44TA, seg_pos={0 0}, upload=475685]: Uploading piece...
Dec 28 05:55:35.062950561 (542h26m34.934033596s) [upload=475685]: Next returned nil segment
Dec 28 05:55:35.062967303 (542h26m34.934050338s) [upload=475685]: Waiting for error group managing segments...
Dec 28 05:55:35.347184408 (542h26m35.218267443s) [noise=true , address=145.79.227.48:4302 , piece_id=2JVUPRBXMI4NZAL6HAKPZUUHCBRJIGBJH4M5WGWNXGVCWTH34XDA, seg_pos={0 0}, upload=475685]: Done uploading piece. err: <nil>
Dec 28 05:55:35.356826297 (542h26m35.227909332s) [noise=true , address=145.79.227.17:4971 , piece_id=7QOHUJVVENFWRXANIKN6MWU4BUCC7ZSAIL4RAF5D4M3UR6Y3HKIA, seg_pos={0 0}, upload=475685]: Done uploading piece. err: <nil>
Dec 28 05:55:35.360595244 (542h26m35.231678289s) [noise=true , address=145.79.227.38:4979 , piece_id=YLHJRB6G2WPOU7RHFGUOON3S3KWXTNIO7BV3T3H7QY6GVHG63EXQ, seg_pos={0 0}, upload=475685]: Done uploading piece. err: <nil>
Dec 28 05:55:35.361069647 (542h26m35.232152682s) [noise=true , address=145.79.227.3:4150 , piece_id=XSN2WUKFGJPAZV7IKKFTVBCE3777W7QJPFINOZSI5DTSHZY2QT2Q, seg_pos={0 0}, upload=475685]: Done uploading piece. err: <nil>
Dec 28 05:55:35.364315959 (542h26m35.235399004s) [noise=true , address=145.79.227.14:4838 , piece_id=GRUDK6C5BUQGJQH7IPC7WFXSRRMTOJJOWOHPHPKVUOK7L5NV3EMA, seg_pos={0 0}, upload=475685]: Done uploading piece. err: <nil>
Dec 28 05:55:35.372200572 (542h26m35.243283607s) [noise=true , address=145.79.227.36:4518 , piece_id=MR22RW6NHOW5UDT6FGZZV6VNZWTUYYLT7QB2GHCWOOOPKHNQ7IQQ, seg_pos={0 0}, upload=475685]: Done uploading piece. err: <nil>
Dec 28 05:55:35.377136982 (542h26m35.248220017s) [noise=true , address=145.79.227.51:4410 , piece_id=LECHFPGJVNJZ42CQSEQ4T24NXHE66P5DECSXXWEKW6GD4AY6S7IQ, seg_pos={0 0}, upload=475685]: Done uploading piece. err: <nil>
Dec 28 05:55:35.377586929 (542h26m35.248669994s) [noise=true , address=145.79.227.4:4187 , piece_id=RFGQBJKERSURKBDIWYC7JBME2N4DITZG3KJLJTEVJF2SZUAWHLWQ, seg_pos={0 0}, upload=475685]: Done uploading piece. err: <nil>
Dec 28 05:55:35.378320015 (542h26m35.24940305s) [noise=true , address=145.79.227.31:4798 , piece_id=TBLHTLG5V5JT5Y737Z77INRR325CH2LNGWZ6YMGCD335H3UHSKRA, seg_pos={0 0}, upload=475685]: Done uploading piece. err: <nil>
Dec 28 05:55:35.744060824 (542h26m35.615143869s) [noise=true , address=45.91.116.7:4012 , piece_id=EUHICAKRSLDYW4HYTXOZTE6OEU4LIEQSWR6NPLPGE4EDZVRPYTGA, seg_pos={0 0}, upload=475685]: Done uploading piece. err: <nil>
Dec 28 05:55:36.337187412 (542h26m36.208270447s) [noise=true , address=77.90.31.11:4553 , piece_id=SMZ5XT7WJD7UIVLG4RXQOG5YAAYKH4A2UAK6FZ3YKY72GLQEP3JA, seg_pos={0 0}, upload=475685]: Done uploading piece. err: <nil>
Dec 28 05:55:37.212533832 (542h26m37.083616877s) [noise=true , address=77.90.31.19:4841 , piece_id=W7EOGATNALKO2EOQIDV5EUVXLQ2XE6I4SKB5K3EXQFVQGIS4OZMQ, seg_pos={0 0}, upload=475685]: Done uploading piece. err: <nil>
Dec 28 05:56:02.402958638 (542h27m2.274041683s) [noise=true , address=145.79.227.19:4079 , piece_id=6YTL65HQ7C66WN74HIJZ45D6ZZUJVNWZLWD3CRVKOBEZST4O5JMA, seg_pos={0 0}, upload=475685]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:35836->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:35836->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35836->145.79.227.19:4079: use of closed network connection
Dec 28 05:56:02.402988726 (542h27m2.274071771s) [noise=true , address=145.79.227.18:4006 , piece_id=2BLQ5ARHTPFXAUXKO7DNPK7QZZXOOYGJASHWZTXD4MGY5PFUEUZA, seg_pos={0 0}, upload=475685]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: write tcp 188.166.225.170:34060->145.79.227.18:4006: use of closed network connection; write tcp 188.166.225.170:34060->145.79.227.18:4006: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34060->145.79.227.18:4006: use of closed network connection
Dec 28 05:56:02.403095910 (542h27m2.274178955s) [noise=true , address=145.79.227.2:4115 , piece_id=QTSI2JB5QPYDE3S4HWTAPGYLYACLT35STUFF2XBX3JAK4XZE44TA, seg_pos={0 0}, upload=475685]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: write tcp 188.166.225.170:37340->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:37340->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37340->145.79.227.2:4115: use of closed network connection
Dec 28 05:56:02.403104256 (542h27m2.274187302s) [noise=true , address=145.79.227.49:4338 , piece_id=2WU4N4PYGZSZXU4QLZFG6EU7ADMND22XWSI7FECE4VMWGXLXU37Q, seg_pos={0 0}, upload=475685]: Done uploading piece. err: ecclient: upload failed (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN, address:145.79.227.49:4338): protocol: write tcp 188.166.225.170:57176->145.79.227.49:4338: use of closed network connection; write tcp 188.166.225.170:57176->145.79.227.49:4338: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57176->145.79.227.49:4338: use of closed network connection
Dec 28 05:56:02.403136398 (542h27m2.274219443s) [seg_pos={0 0}, upload=475685]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 28 05:56:02.693153827 (542h27m2.564236872s) [upload=475685]: Done uploading segments...
Dec 28 05:56:02.693202830 (542h27m2.564285875s) [upload=475685]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.989238069s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [.................X.........X...X....................X.......] duration:
55.433256943s trace:
https://jaeger.dev.storj.io/trace/35a88c4b3a3d78b6
FAIL (2025-12-28 05:57:03 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH, Piece ID: HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA) context deadline exceeded; piecestore: (Node ID: 1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, Piece ID: GY4LXIIIB7SVWA2WRAEHJVB6HLN6XQYISF7ZPVE4KA42C23MEQ4A) context deadline exceeded; piecestore: (Node ID: 1RQ3vtArvKAvaMJTVa9VDDVdWR5oBuZ7gaPB3pzc1MouqBZhWu, Piece ID: FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ) context deadline exceeded; piecestore: (Node ID: 127KQ2X6VXddEeFRMau7MVGwtTsUmFsm8hTYDR2A6B4A1oVKfyo, Piece ID: V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ) context deadline exceeded
======LOGS======
) [noise=true , address=145.79.227.44:4209 , piece_id=JRNDKXGHFY5OJYSY4ESURYMK5BNRDWY6INLHOWEK55KHMNRJO4AQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.655023976 (542h19m51.526107021s) [noise=true , address=77.90.31.13:4647 , piece_id=SOPU7ZUZQXRLU6TUL4L3ODX7U7WFAKUDOSK5UO66WMAIANU6PVUA, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.655676448 (542h19m51.526759493s) [noise=true , address=45.91.116.8:4059 , piece_id=OSA2YK4AH2DPQBNSJDFT3PLTGPJM6N7EOB7X27ZP2WM52HWSHT4Q, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.656204463 (542h19m51.527287509s) [noise=true , address=145.79.227.49:4370 , piece_id=QDMDBC3576X7ZYGUAC2DLJKWBNVOLSROCMA4UNFDZ47TTADILUAQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.656699927 (542h19m51.527782972s) [noise=true , address=45.91.116.9:4290 , piece_id=MP5DAA5UGHWQ3BA5R7GBWNIV2B4JYCL5WUSV2PY7GTO4PNLH2F2Q, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.657183808 (542h19m51.528266883s) [noise=true , address=77.90.31.18:4827 , piece_id=V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.657682177 (542h19m51.528765222s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.658706768 (542h19m51.529789813s) [noise=true , address=145.79.227.33:4876 , piece_id=E5QQDATADJON5NOOQ6X37GFKUQCFUKJRGK6G2EURBSUJVLF72Y7A, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.660113707 (542h19m51.531196752s) [noise=true , address=145.79.227.34:4915 , piece_id=MR5WOODFPXH57I6O2JCMC3EE5OWXF23CFLLATZXFP5XK6YPWDZJA, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.661356113 (542h19m51.532439158s) [noise=true , address=77.90.31.15:4727 , piece_id=3ACH3RUAR5KUXZU23AL3OQSGIZQOU5MFOFRN3JUYLQJBJCA4JOZQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.664431910 (542h19m51.535514955s) [noise=true , address=45.91.116.6:4998 , piece_id=FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.666377204 (542h19m51.537460249s) [noise=true , address=145.79.227.40:4067 , piece_id=36Q7DHHN5ELVD3BYSNUGJ7WYXNGIDKERTOPRAZJNJ7YQTOUPWEYQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.666754502 (542h19m51.537837548s) [noise=true , address=77.90.31.12:4594 , piece_id=DNXKJQOUHLAWMOCR35THP5CKCB6LZBW4GMJU6O6KKVSIJ3ALLONQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.668451284 (542h19m51.539534319s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.671076552 (542h19m51.542159587s) [noise=true , address=77.90.31.11:4560 , piece_id=HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.674568571 (542h19m51.545651626s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:52.998088476 (542h19m52.869171521s) [upload=475588]: Next returned nil segment
Dec 28 05:48:52.998103846 (542h19m52.869186881s) [upload=475588]: Waiting for error group managing segments...
Dec 28 05:48:53.543654582 (542h19m53.414737627s) [noise=true , address=145.79.227.40:4067 , piece_id=36Q7DHHN5ELVD3BYSNUGJ7WYXNGIDKERTOPRAZJNJ7YQTOUPWEYQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.543744133 (542h19m53.414827178s) [noise=true , address=145.79.227.49:4370 , piece_id=QDMDBC3576X7ZYGUAC2DLJKWBNVOLSROCMA4UNFDZ47TTADILUAQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.556254274 (542h19m53.427337309s) [noise=true , address=145.79.227.34:4915 , piece_id=MR5WOODFPXH57I6O2JCMC3EE5OWXF23CFLLATZXFP5XK6YPWDZJA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.810781113 (542h19m53.681864158s) [noise=true , address=77.90.31.13:4647 , piece_id=SOPU7ZUZQXRLU6TUL4L3ODX7U7WFAKUDOSK5UO66WMAIANU6PVUA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.814762255 (542h19m53.6858453s) [noise=true , address=77.90.31.16:4756 , piece_id=GY4LXIIIB7SVWA2WRAEHJVB6HLN6XQYISF7ZPVE4KA42C23MEQ4A, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.816966662 (542h19m53.688049707s) [noise=true , address=77.90.31.15:4727 , piece_id=3ACH3RUAR5KUXZU23AL3OQSGIZQOU5MFOFRN3JUYLQJBJCA4JOZQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.895077718 (542h19m53.766160783s) [noise=true , address=45.91.116.9:4290 , piece_id=MP5DAA5UGHWQ3BA5R7GBWNIV2B4JYCL5WUSV2PY7GTO4PNLH2F2Q, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.906586933 (542h19m53.777669978s) [noise=true , address=45.91.116.8:4059 , piece_id=OSA2YK4AH2DPQBNSJDFT3PLTGPJM6N7EOB7X27ZP2WM52HWSHT4Q, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.930785842 (542h19m53.801868877s) [noise=true , address=145.79.227.33:4876 , piece_id=E5QQDATADJON5NOOQ6X37GFKUQCFUKJRGK6G2EURBSUJVLF72Y7A, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.087446700 (542h19m53.958529735s) [noise=true , address=45.91.116.6:4998 , piece_id=FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.129981903 (542h19m54.001064938s) [noise=true , address=77.90.31.11:4560 , piece_id=HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.264740966 (542h19m54.135824011s) [noise=true , address=77.90.31.12:4594 , piece_id=DNXKJQOUHLAWMOCR35THP5CKCB6LZBW4GMJU6O6KKVSIJ3ALLONQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.368348401 (542h19m54.239431446s) [noise=true , address=77.90.31.18:4827 , piece_id=V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.437713885 (542h19m54.30879693s) [noise=true , address=145.79.227.44:4209 , piece_id=JRNDKXGHFY5OJYSY4ESURYMK5BNRDWY6INLHOWEK55KHMNRJO4AQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.437828132 (542h19m54.308911187s) [seg_pos={0 0}, upload=475588]: Segment reached optimal threshold of 14 pieces.
Dec 28 05:48:54.438064122 (542h19m54.309147167s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF
Dec 28 05:48:54.438074481 (542h19m54.309157546s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold
Dec 28 05:48:54.438099619 (542h19m54.309182664s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F4fJAGJM2ZPRt7vJH2cZmEbYeujmxcfi7aPZ2g9VPTNYBYRAi): protocol: expected piece hash; context canceled; EOF
Dec 28 05:48:54.438108586 (542h19m54.309191631s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold
Dec 28 05:48:54.438243403 (542h19m54.309326458s) [noise=true , address=77.90.31.19:4876 , piece_id=PUIQX7IZSNPKZTZTYH32KTFUM7K53KWZASLPT6JHAPMOB2JCXPGA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12H3UNBGeAufYSdVvEgYkv2QWeFtjKMvHvYGeqGeBvjEHqDDjBB): protocol: expected piece hash; context canceled; EOF
Dec 28 05:48:54.438252169 (542h19m54.309335214s) [noise=true , address=77.90.31.19:4876 , piece_id=PUIQX7IZSNPKZTZTYH32KTFUM7K53KWZASLPT6JHAPMOB2JCXPGA, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold
Dec 28 05:48:54.438391024 (542h19m54.309474069s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection
Dec 28 05:48:54.438398588 (542h19m54.309481633s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold
Dec 28 05:48:54.438699081 (542h19m54.309782126s) [seg_pos={0 0}, upload=475588]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 05:48:54.438794753 (542h19m54.309877798s) [upload=475588]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 05:48:54.438801255 (542h19m54.30988429s) [upload=475588]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 05:48:54.728435695 (542h19m54.59951876s) [upload=475588]: Done uploading segments...
Dec 28 05:48:54.728457938 (542h19m54.599540983s) [upload=475588]: Upload finished. err: <nil>
Duration: 55.433285105s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
20 out of 60 is failed (threshold=3) [.XX.X.XX..XX.X...X.X.XX....X...X.X..X.....X........X...X..X.] duration:
5.828472016s trace:
https://jaeger.dev.storj.io/trace/2df8490696898dfd
FAIL (2025-12-28 05:56:48 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 28 05:55:33.093317793 (542h26m32.964400828s) [upload=475685]: Starting upload
Dec 28 05:55:33.093367507 (542h26m32.964450542s) [upload=475685]: Done starting upload
Dec 28 05:55:33.093536659 (542h26m32.964619694s) [upload=475685]: Uploading segments...
Dec 28 05:55:33.093569472 (542h26m32.964652507s) [upload=475685]: Got next segment. Inline: false
Dec 28 05:55:33.093593899 (542h26m32.964676934s) [upload=475685]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 05:55:33.093597686 (542h26m32.964680721s) [upload=475685]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 05:55:33.339343235 (542h26m33.21042627s) [seg_pos={0 0}, upload=475685]: Begin upload segment...
Dec 28 05:55:33.339391968 (542h26m33.210475003s) [seg_pos={0 0}, upload=475685]: Done begin upload segment.
Dec 28 05:55:33.339427986 (542h26m33.210511031s) [noise=true , address=145.79.227.4:4187 , piece_id=RFGQBJKERSURKBDIWYC7JBME2N4DITZG3KJLJTEVJF2SZUAWHLWQ, seg_pos={0 0}, upload=475685]: Uploading piece...
Dec 28 05:55:33.340124412 (542h26m33.211207447s) [noise=true , address=145.79.227.18:4006 , piece_id=2BLQ5ARHTPFXAUXKO7DNPK7QZZXOOYGJASHWZTXD4MGY5PFUEUZA, seg_pos={0 0}, upload=475685]: Uploading piece...
Dec 28 05:55:33.340447347 (542h26m33.211530382s) [noise=true , address=145.79.227.14:4838 , piece_id=GRUDK6C5BUQGJQH7IPC7WFXSRRMTOJJOWOHPHPKVUOK7L5NV3EMA, seg_pos={0 0}, upload=475685]: Uploading piece...
Dec 28 05:55:33.340838633 (542h26m33.211921668s) [noise=true , address=77.90.31.11:4553 , piece_id=SMZ5XT7WJD7UIVLG4RXQOG5YAAYKH4A2UAK6FZ3YKY72GLQEP3JA, seg_pos={0 0}, upload=475685]: Uploading piece...
Dec 28 05:55:33.341125900 (542h26m33.212208935s) [noise=true , address=145.79.227.51:4410 , piece_id=LECHFPGJVNJZ42CQSEQ4T24NXHE66P5DECSXXWEKW6GD4AY6S7IQ, seg_pos={0 0}, upload=475685]: Uploading piece...
Dec 28 05:55:33.341419378 (542h26m33.212502413s) [noise=true , address=77.90.31.19:4841 , piece_id=W7EOGATNALKO2EOQIDV5EUVXLQ2XE6I4SKB5K3EXQFVQGIS4OZMQ, seg_pos={0 0}, upload=475685]: Uploading piece...
Dec 28 05:55:33.341732895 (542h26m33.21281593s) [noise=true , address=145.79.227.19:4079 , piece_id=6YTL65HQ7C66WN74HIJZ45D6ZZUJVNWZLWD3CRVKOBEZST4O5JMA, seg_pos={0 0}, upload=475685]: Uploading piece...
Dec 28 05:55:33.342080246 (542h26m33.213163292s) [noise=true , address=145.79.227.3:4150 , piece_id=XSN2WUKFGJPAZV7IKKFTVBCE3777W7QJPFINOZSI5DTSHZY2QT2Q, seg_pos={0 0}, upload=475685]: Uploading piece...
Dec 28 05:55:33.342590188 (542h26m33.213673233s) [noise=true , address=145.79.227.31:4798 , piece_id=TBLHTLG5V5JT5Y737Z77INRR325CH2LNGWZ6YMGCD335H3UHSKRA, seg_pos={0 0}, upload=475685]: Uploading piece...
Dec 28 05:55:33.343775314 (542h26m33.21485835s) [noise=true , address=145.79.227.17:4971 , piece_id=7QOHUJVVENFWRXANIKN6MWU4BUCC7ZSAIL4RAF5D4M3UR6Y3HKIA, seg_pos={0 0}, upload=475685]: Uploading piece...
Dec 28 05:55:33.344875540 (542h26m33.215958575s) [noise=true , address=145.79.227.49:4338 , piece_id=2WU4N4PYGZSZXU4QLZFG6EU7ADMND22XWSI7FECE4VMWGXLXU37Q, seg_pos={0 0}, upload=475685]: Uploading piece...
Dec 28 05:55:33.346640229 (542h26m33.217723265s) [noise=true , address=145.79.227.38:4979 , piece_id=YLHJRB6G2WPOU7RHFGUOON3S3KWXTNIO7BV3T3H7QY6GVHG63EXQ, seg_pos={0 0}, upload=475685]: Uploading piece...
Dec 28 05:55:33.347952999 (542h26m33.219036035s) [noise=true , address=45.91.116.7:4012 , piece_id=EUHICAKRSLDYW4HYTXOZTE6OEU4LIEQSWR6NPLPGE4EDZVRPYTGA, seg_pos={0 0}, upload=475685]: Uploading piece...
Dec 28 05:55:33.349149938 (542h26m33.220232973s) [noise=true , address=145.79.227.36:4518 , piece_id=MR22RW6NHOW5UDT6FGZZV6VNZWTUYYLT7QB2GHCWOOOPKHNQ7IQQ, seg_pos={0 0}, upload=475685]: Uploading piece...
Dec 28 05:55:33.349408171 (542h26m33.220491206s) [noise=true , address=145.79.227.48:4302 , piece_id=2JVUPRBXMI4NZAL6HAKPZUUHCBRJIGBJH4M5WGWNXGVCWTH34XDA, seg_pos={0 0}, upload=475685]: Uploading piece...
Dec 28 05:55:33.350929607 (542h26m33.222012662s) [noise=true , address=145.79.227.2:4115 , piece_id=QTSI2JB5QPYDE3S4HWTAPGYLYACLT35STUFF2XBX3JAK4XZE44TA, seg_pos={0 0}, upload=475685]: Uploading piece...
Dec 28 05:55:35.062950561 (542h26m34.934033596s) [upload=475685]: Next returned nil segment
Dec 28 05:55:35.062967303 (542h26m34.934050338s) [upload=475685]: Waiting for error group managing segments...
Dec 28 05:55:35.347184408 (542h26m35.218267443s) [noise=true , address=145.79.227.48:4302 , piece_id=2JVUPRBXMI4NZAL6HAKPZUUHCBRJIGBJH4M5WGWNXGVCWTH34XDA, seg_pos={0 0}, upload=475685]: Done uploading piece. err: <nil>
Dec 28 05:55:35.356826297 (542h26m35.227909332s) [noise=true , address=145.79.227.17:4971 , piece_id=7QOHUJVVENFWRXANIKN6MWU4BUCC7ZSAIL4RAF5D4M3UR6Y3HKIA, seg_pos={0 0}, upload=475685]: Done uploading piece. err: <nil>
Dec 28 05:55:35.360595244 (542h26m35.231678289s) [noise=true , address=145.79.227.38:4979 , piece_id=YLHJRB6G2WPOU7RHFGUOON3S3KWXTNIO7BV3T3H7QY6GVHG63EXQ, seg_pos={0 0}, upload=475685]: Done uploading piece. err: <nil>
Dec 28 05:55:35.361069647 (542h26m35.232152682s) [noise=true , address=145.79.227.3:4150 , piece_id=XSN2WUKFGJPAZV7IKKFTVBCE3777W7QJPFINOZSI5DTSHZY2QT2Q, seg_pos={0 0}, upload=475685]: Done uploading piece. err: <nil>
Dec 28 05:55:35.364315959 (542h26m35.235399004s) [noise=true , address=145.79.227.14:4838 , piece_id=GRUDK6C5BUQGJQH7IPC7WFXSRRMTOJJOWOHPHPKVUOK7L5NV3EMA, seg_pos={0 0}, upload=475685]: Done uploading piece. err: <nil>
Dec 28 05:55:35.372200572 (542h26m35.243283607s) [noise=true , address=145.79.227.36:4518 , piece_id=MR22RW6NHOW5UDT6FGZZV6VNZWTUYYLT7QB2GHCWOOOPKHNQ7IQQ, seg_pos={0 0}, upload=475685]: Done uploading piece. err: <nil>
Dec 28 05:55:35.377136982 (542h26m35.248220017s) [noise=true , address=145.79.227.51:4410 , piece_id=LECHFPGJVNJZ42CQSEQ4T24NXHE66P5DECSXXWEKW6GD4AY6S7IQ, seg_pos={0 0}, upload=475685]: Done uploading piece. err: <nil>
Dec 28 05:55:35.377586929 (542h26m35.248669994s) [noise=true , address=145.79.227.4:4187 , piece_id=RFGQBJKERSURKBDIWYC7JBME2N4DITZG3KJLJTEVJF2SZUAWHLWQ, seg_pos={0 0}, upload=475685]: Done uploading piece. err: <nil>
Dec 28 05:55:35.378320015 (542h26m35.24940305s) [noise=true , address=145.79.227.31:4798 , piece_id=TBLHTLG5V5JT5Y737Z77INRR325CH2LNGWZ6YMGCD335H3UHSKRA, seg_pos={0 0}, upload=475685]: Done uploading piece. err: <nil>
Dec 28 05:55:35.744060824 (542h26m35.615143869s) [noise=true , address=45.91.116.7:4012 , piece_id=EUHICAKRSLDYW4HYTXOZTE6OEU4LIEQSWR6NPLPGE4EDZVRPYTGA, seg_pos={0 0}, upload=475685]: Done uploading piece. err: <nil>
Dec 28 05:55:36.337187412 (542h26m36.208270447s) [noise=true , address=77.90.31.11:4553 , piece_id=SMZ5XT7WJD7UIVLG4RXQOG5YAAYKH4A2UAK6FZ3YKY72GLQEP3JA, seg_pos={0 0}, upload=475685]: Done uploading piece. err: <nil>
Dec 28 05:55:37.212533832 (542h26m37.083616877s) [noise=true , address=77.90.31.19:4841 , piece_id=W7EOGATNALKO2EOQIDV5EUVXLQ2XE6I4SKB5K3EXQFVQGIS4OZMQ, seg_pos={0 0}, upload=475685]: Done uploading piece. err: <nil>
Dec 28 05:56:02.402958638 (542h27m2.274041683s) [noise=true , address=145.79.227.19:4079 , piece_id=6YTL65HQ7C66WN74HIJZ45D6ZZUJVNWZLWD3CRVKOBEZST4O5JMA, seg_pos={0 0}, upload=475685]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:35836->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:35836->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35836->145.79.227.19:4079: use of closed network connection
Dec 28 05:56:02.402988726 (542h27m2.274071771s) [noise=true , address=145.79.227.18:4006 , piece_id=2BLQ5ARHTPFXAUXKO7DNPK7QZZXOOYGJASHWZTXD4MGY5PFUEUZA, seg_pos={0 0}, upload=475685]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: write tcp 188.166.225.170:34060->145.79.227.18:4006: use of closed network connection; write tcp 188.166.225.170:34060->145.79.227.18:4006: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34060->145.79.227.18:4006: use of closed network connection
Dec 28 05:56:02.403095910 (542h27m2.274178955s) [noise=true , address=145.79.227.2:4115 , piece_id=QTSI2JB5QPYDE3S4HWTAPGYLYACLT35STUFF2XBX3JAK4XZE44TA, seg_pos={0 0}, upload=475685]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: write tcp 188.166.225.170:37340->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:37340->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37340->145.79.227.2:4115: use of closed network connection
Dec 28 05:56:02.403104256 (542h27m2.274187302s) [noise=true , address=145.79.227.49:4338 , piece_id=2WU4N4PYGZSZXU4QLZFG6EU7ADMND22XWSI7FECE4VMWGXLXU37Q, seg_pos={0 0}, upload=475685]: Done uploading piece. err: ecclient: upload failed (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN, address:145.79.227.49:4338): protocol: write tcp 188.166.225.170:57176->145.79.227.49:4338: use of closed network connection; write tcp 188.166.225.170:57176->145.79.227.49:4338: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57176->145.79.227.49:4338: use of closed network connection
Dec 28 05:56:02.403136398 (542h27m2.274219443s) [seg_pos={0 0}, upload=475685]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 28 05:56:02.693153827 (542h27m2.564236872s) [upload=475685]: Done uploading segments...
Dec 28 05:56:02.693202830 (542h27m2.564285875s) [upload=475685]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 5.828498246s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
25 out of 60 is failed (threshold=6) [..XXX....XXXX....XXX.X...X....X...XX.X..XXX....X..XX.X...X.X] duration:
5.000870052s trace:
https://jaeger.dev.storj.io/trace/444fe9e1aa1340ed
FAIL (2025-12-28 05:56:45 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.00089046s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
24 out of 60 is failed (threshold=6) [...XXX....XXXX....XXX.X...X....X...XX.X..XXX....X..XX.X...X.] duration:
3.562048133s trace:
https://jaeger.dev.storj.io/trace/5562ed0e741027df
FAIL (2025-12-28 05:56:16 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.562074213s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
21 out of 60 is failed (threshold=3) [X.XX.X.XX..XX.X...X.X.XX....X...X.X..X.....X........X...X..X] duration:
30.291979626s trace:
https://jaeger.dev.storj.io/trace/2df8490696898dfd
FAIL (2025-12-28 05:56:02 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 12)
======LOGS======
Dec 28 05:55:33.093317793 (542h26m32.964400828s) [upload=475685]: Starting upload
Dec 28 05:55:33.093367507 (542h26m32.964450542s) [upload=475685]: Done starting upload
Dec 28 05:55:33.093536659 (542h26m32.964619694s) [upload=475685]: Uploading segments...
Dec 28 05:55:33.093569472 (542h26m32.964652507s) [upload=475685]: Got next segment. Inline: false
Dec 28 05:55:33.093593899 (542h26m32.964676934s) [upload=475685]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 05:55:33.093597686 (542h26m32.964680721s) [upload=475685]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 05:55:33.339343235 (542h26m33.21042627s) [seg_pos={0 0}, upload=475685]: Begin upload segment...
Dec 28 05:55:33.339391968 (542h26m33.210475003s) [seg_pos={0 0}, upload=475685]: Done begin upload segment.
Dec 28 05:55:33.339427986 (542h26m33.210511031s) [noise=true , address=145.79.227.4:4187 , piece_id=RFGQBJKERSURKBDIWYC7JBME2N4DITZG3KJLJTEVJF2SZUAWHLWQ, seg_pos={0 0}, upload=475685]: Uploading piece...
Dec 28 05:55:33.340124412 (542h26m33.211207447s) [noise=true , address=145.79.227.18:4006 , piece_id=2BLQ5ARHTPFXAUXKO7DNPK7QZZXOOYGJASHWZTXD4MGY5PFUEUZA, seg_pos={0 0}, upload=475685]: Uploading piece...
Dec 28 05:55:33.340447347 (542h26m33.211530382s) [noise=true , address=145.79.227.14:4838 , piece_id=GRUDK6C5BUQGJQH7IPC7WFXSRRMTOJJOWOHPHPKVUOK7L5NV3EMA, seg_pos={0 0}, upload=475685]: Uploading piece...
Dec 28 05:55:33.340838633 (542h26m33.211921668s) [noise=true , address=77.90.31.11:4553 , piece_id=SMZ5XT7WJD7UIVLG4RXQOG5YAAYKH4A2UAK6FZ3YKY72GLQEP3JA, seg_pos={0 0}, upload=475685]: Uploading piece...
Dec 28 05:55:33.341125900 (542h26m33.212208935s) [noise=true , address=145.79.227.51:4410 , piece_id=LECHFPGJVNJZ42CQSEQ4T24NXHE66P5DECSXXWEKW6GD4AY6S7IQ, seg_pos={0 0}, upload=475685]: Uploading piece...
Dec 28 05:55:33.341419378 (542h26m33.212502413s) [noise=true , address=77.90.31.19:4841 , piece_id=W7EOGATNALKO2EOQIDV5EUVXLQ2XE6I4SKB5K3EXQFVQGIS4OZMQ, seg_pos={0 0}, upload=475685]: Uploading piece...
Dec 28 05:55:33.341732895 (542h26m33.21281593s) [noise=true , address=145.79.227.19:4079 , piece_id=6YTL65HQ7C66WN74HIJZ45D6ZZUJVNWZLWD3CRVKOBEZST4O5JMA, seg_pos={0 0}, upload=475685]: Uploading piece...
Dec 28 05:55:33.342080246 (542h26m33.213163292s) [noise=true , address=145.79.227.3:4150 , piece_id=XSN2WUKFGJPAZV7IKKFTVBCE3777W7QJPFINOZSI5DTSHZY2QT2Q, seg_pos={0 0}, upload=475685]: Uploading piece...
Dec 28 05:55:33.342590188 (542h26m33.213673233s) [noise=true , address=145.79.227.31:4798 , piece_id=TBLHTLG5V5JT5Y737Z77INRR325CH2LNGWZ6YMGCD335H3UHSKRA, seg_pos={0 0}, upload=475685]: Uploading piece...
Dec 28 05:55:33.343775314 (542h26m33.21485835s) [noise=true , address=145.79.227.17:4971 , piece_id=7QOHUJVVENFWRXANIKN6MWU4BUCC7ZSAIL4RAF5D4M3UR6Y3HKIA, seg_pos={0 0}, upload=475685]: Uploading piece...
Dec 28 05:55:33.344875540 (542h26m33.215958575s) [noise=true , address=145.79.227.49:4338 , piece_id=2WU4N4PYGZSZXU4QLZFG6EU7ADMND22XWSI7FECE4VMWGXLXU37Q, seg_pos={0 0}, upload=475685]: Uploading piece...
Dec 28 05:55:33.346640229 (542h26m33.217723265s) [noise=true , address=145.79.227.38:4979 , piece_id=YLHJRB6G2WPOU7RHFGUOON3S3KWXTNIO7BV3T3H7QY6GVHG63EXQ, seg_pos={0 0}, upload=475685]: Uploading piece...
Dec 28 05:55:33.347952999 (542h26m33.219036035s) [noise=true , address=45.91.116.7:4012 , piece_id=EUHICAKRSLDYW4HYTXOZTE6OEU4LIEQSWR6NPLPGE4EDZVRPYTGA, seg_pos={0 0}, upload=475685]: Uploading piece...
Dec 28 05:55:33.349149938 (542h26m33.220232973s) [noise=true , address=145.79.227.36:4518 , piece_id=MR22RW6NHOW5UDT6FGZZV6VNZWTUYYLT7QB2GHCWOOOPKHNQ7IQQ, seg_pos={0 0}, upload=475685]: Uploading piece...
Dec 28 05:55:33.349408171 (542h26m33.220491206s) [noise=true , address=145.79.227.48:4302 , piece_id=2JVUPRBXMI4NZAL6HAKPZUUHCBRJIGBJH4M5WGWNXGVCWTH34XDA, seg_pos={0 0}, upload=475685]: Uploading piece...
Dec 28 05:55:33.350929607 (542h26m33.222012662s) [noise=true , address=145.79.227.2:4115 , piece_id=QTSI2JB5QPYDE3S4HWTAPGYLYACLT35STUFF2XBX3JAK4XZE44TA, seg_pos={0 0}, upload=475685]: Uploading piece...
Dec 28 05:55:35.062950561 (542h26m34.934033596s) [upload=475685]: Next returned nil segment
Dec 28 05:55:35.062967303 (542h26m34.934050338s) [upload=475685]: Waiting for error group managing segments...
Dec 28 05:55:35.347184408 (542h26m35.218267443s) [noise=true , address=145.79.227.48:4302 , piece_id=2JVUPRBXMI4NZAL6HAKPZUUHCBRJIGBJH4M5WGWNXGVCWTH34XDA, seg_pos={0 0}, upload=475685]: Done uploading piece. err: <nil>
Dec 28 05:55:35.356826297 (542h26m35.227909332s) [noise=true , address=145.79.227.17:4971 , piece_id=7QOHUJVVENFWRXANIKN6MWU4BUCC7ZSAIL4RAF5D4M3UR6Y3HKIA, seg_pos={0 0}, upload=475685]: Done uploading piece. err: <nil>
Dec 28 05:55:35.360595244 (542h26m35.231678289s) [noise=true , address=145.79.227.38:4979 , piece_id=YLHJRB6G2WPOU7RHFGUOON3S3KWXTNIO7BV3T3H7QY6GVHG63EXQ, seg_pos={0 0}, upload=475685]: Done uploading piece. err: <nil>
Dec 28 05:55:35.361069647 (542h26m35.232152682s) [noise=true , address=145.79.227.3:4150 , piece_id=XSN2WUKFGJPAZV7IKKFTVBCE3777W7QJPFINOZSI5DTSHZY2QT2Q, seg_pos={0 0}, upload=475685]: Done uploading piece. err: <nil>
Dec 28 05:55:35.364315959 (542h26m35.235399004s) [noise=true , address=145.79.227.14:4838 , piece_id=GRUDK6C5BUQGJQH7IPC7WFXSRRMTOJJOWOHPHPKVUOK7L5NV3EMA, seg_pos={0 0}, upload=475685]: Done uploading piece. err: <nil>
Dec 28 05:55:35.372200572 (542h26m35.243283607s) [noise=true , address=145.79.227.36:4518 , piece_id=MR22RW6NHOW5UDT6FGZZV6VNZWTUYYLT7QB2GHCWOOOPKHNQ7IQQ, seg_pos={0 0}, upload=475685]: Done uploading piece. err: <nil>
Dec 28 05:55:35.377136982 (542h26m35.248220017s) [noise=true , address=145.79.227.51:4410 , piece_id=LECHFPGJVNJZ42CQSEQ4T24NXHE66P5DECSXXWEKW6GD4AY6S7IQ, seg_pos={0 0}, upload=475685]: Done uploading piece. err: <nil>
Dec 28 05:55:35.377586929 (542h26m35.248669994s) [noise=true , address=145.79.227.4:4187 , piece_id=RFGQBJKERSURKBDIWYC7JBME2N4DITZG3KJLJTEVJF2SZUAWHLWQ, seg_pos={0 0}, upload=475685]: Done uploading piece. err: <nil>
Dec 28 05:55:35.378320015 (542h26m35.24940305s) [noise=true , address=145.79.227.31:4798 , piece_id=TBLHTLG5V5JT5Y737Z77INRR325CH2LNGWZ6YMGCD335H3UHSKRA, seg_pos={0 0}, upload=475685]: Done uploading piece. err: <nil>
Dec 28 05:55:35.744060824 (542h26m35.615143869s) [noise=true , address=45.91.116.7:4012 , piece_id=EUHICAKRSLDYW4HYTXOZTE6OEU4LIEQSWR6NPLPGE4EDZVRPYTGA, seg_pos={0 0}, upload=475685]: Done uploading piece. err: <nil>
Dec 28 05:55:36.337187412 (542h26m36.208270447s) [noise=true , address=77.90.31.11:4553 , piece_id=SMZ5XT7WJD7UIVLG4RXQOG5YAAYKH4A2UAK6FZ3YKY72GLQEP3JA, seg_pos={0 0}, upload=475685]: Done uploading piece. err: <nil>
Dec 28 05:55:37.212533832 (542h26m37.083616877s) [noise=true , address=77.90.31.19:4841 , piece_id=W7EOGATNALKO2EOQIDV5EUVXLQ2XE6I4SKB5K3EXQFVQGIS4OZMQ, seg_pos={0 0}, upload=475685]: Done uploading piece. err: <nil>
Dec 28 05:56:02.402958638 (542h27m2.274041683s) [noise=true , address=145.79.227.19:4079 , piece_id=6YTL65HQ7C66WN74HIJZ45D6ZZUJVNWZLWD3CRVKOBEZST4O5JMA, seg_pos={0 0}, upload=475685]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:145.79.227.19:4079): protocol: write tcp 188.166.225.170:35836->145.79.227.19:4079: use of closed network connection; write tcp 188.166.225.170:35836->145.79.227.19:4079: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:35836->145.79.227.19:4079: use of closed network connection
Dec 28 05:56:02.402988726 (542h27m2.274071771s) [noise=true , address=145.79.227.18:4006 , piece_id=2BLQ5ARHTPFXAUXKO7DNPK7QZZXOOYGJASHWZTXD4MGY5PFUEUZA, seg_pos={0 0}, upload=475685]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:145.79.227.18:4006): protocol: write tcp 188.166.225.170:34060->145.79.227.18:4006: use of closed network connection; write tcp 188.166.225.170:34060->145.79.227.18:4006: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34060->145.79.227.18:4006: use of closed network connection
Dec 28 05:56:02.403095910 (542h27m2.274178955s) [noise=true , address=145.79.227.2:4115 , piece_id=QTSI2JB5QPYDE3S4HWTAPGYLYACLT35STUFF2XBX3JAK4XZE44TA, seg_pos={0 0}, upload=475685]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: write tcp 188.166.225.170:37340->145.79.227.2:4115: use of closed network connection; write tcp 188.166.225.170:37340->145.79.227.2:4115: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37340->145.79.227.2:4115: use of closed network connection
Dec 28 05:56:02.403104256 (542h27m2.274187302s) [noise=true , address=145.79.227.49:4338 , piece_id=2WU4N4PYGZSZXU4QLZFG6EU7ADMND22XWSI7FECE4VMWGXLXU37Q, seg_pos={0 0}, upload=475685]: Done uploading piece. err: ecclient: upload failed (node:1cTadUV1Zu2Z8MoobXvyZ965fv6jPFhYGsy5qgRZHgWeSwyitN, address:145.79.227.49:4338): protocol: write tcp 188.166.225.170:57176->145.79.227.49:4338: use of closed network connection; write tcp 188.166.225.170:57176->145.79.227.49:4338: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:57176->145.79.227.49:4338: use of closed network connection
Dec 28 05:56:02.403136398 (542h27m2.274219443s) [seg_pos={0 0}, upload=475685]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil>
Dec 28 05:56:02.693153827 (542h27m2.564236872s) [upload=475685]: Done uploading segments...
Dec 28 05:56:02.693202830 (542h27m2.564285875s) [upload=475685]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12)
Duration: 30.292001298s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
24 out of 60 is failed (threshold=6) [....XXX....XXXX....XXX.X...X....X...XX.X..XXX....X..XX.X...X] duration:
5.00085395s trace:
https://jaeger.dev.storj.io/trace/5562ed0e741027df
FAIL (2025-12-28 05:55:53 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000892362s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [..................X.........X...X....................X......] duration:
42.100201379s trace:
https://jaeger.dev.storj.io/trace/35a88c4b3a3d78b6
FAIL (2025-12-28 05:55:38 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH, Piece ID: HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA) context deadline exceeded; piecestore: (Node ID: 1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, Piece ID: GY4LXIIIB7SVWA2WRAEHJVB6HLN6XQYISF7ZPVE4KA42C23MEQ4A) context deadline exceeded; piecestore: (Node ID: 1RQ3vtArvKAvaMJTVa9VDDVdWR5oBuZ7gaPB3pzc1MouqBZhWu, Piece ID: FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ) context deadline exceeded; piecestore: (Node ID: 127KQ2X6VXddEeFRMau7MVGwtTsUmFsm8hTYDR2A6B4A1oVKfyo, Piece ID: V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ) context deadline exceeded
======LOGS======
) [noise=true , address=145.79.227.44:4209 , piece_id=JRNDKXGHFY5OJYSY4ESURYMK5BNRDWY6INLHOWEK55KHMNRJO4AQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.655023976 (542h19m51.526107021s) [noise=true , address=77.90.31.13:4647 , piece_id=SOPU7ZUZQXRLU6TUL4L3ODX7U7WFAKUDOSK5UO66WMAIANU6PVUA, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.655676448 (542h19m51.526759493s) [noise=true , address=45.91.116.8:4059 , piece_id=OSA2YK4AH2DPQBNSJDFT3PLTGPJM6N7EOB7X27ZP2WM52HWSHT4Q, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.656204463 (542h19m51.527287509s) [noise=true , address=145.79.227.49:4370 , piece_id=QDMDBC3576X7ZYGUAC2DLJKWBNVOLSROCMA4UNFDZ47TTADILUAQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.656699927 (542h19m51.527782972s) [noise=true , address=45.91.116.9:4290 , piece_id=MP5DAA5UGHWQ3BA5R7GBWNIV2B4JYCL5WUSV2PY7GTO4PNLH2F2Q, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.657183808 (542h19m51.528266883s) [noise=true , address=77.90.31.18:4827 , piece_id=V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.657682177 (542h19m51.528765222s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.658706768 (542h19m51.529789813s) [noise=true , address=145.79.227.33:4876 , piece_id=E5QQDATADJON5NOOQ6X37GFKUQCFUKJRGK6G2EURBSUJVLF72Y7A, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.660113707 (542h19m51.531196752s) [noise=true , address=145.79.227.34:4915 , piece_id=MR5WOODFPXH57I6O2JCMC3EE5OWXF23CFLLATZXFP5XK6YPWDZJA, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.661356113 (542h19m51.532439158s) [noise=true , address=77.90.31.15:4727 , piece_id=3ACH3RUAR5KUXZU23AL3OQSGIZQOU5MFOFRN3JUYLQJBJCA4JOZQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.664431910 (542h19m51.535514955s) [noise=true , address=45.91.116.6:4998 , piece_id=FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.666377204 (542h19m51.537460249s) [noise=true , address=145.79.227.40:4067 , piece_id=36Q7DHHN5ELVD3BYSNUGJ7WYXNGIDKERTOPRAZJNJ7YQTOUPWEYQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.666754502 (542h19m51.537837548s) [noise=true , address=77.90.31.12:4594 , piece_id=DNXKJQOUHLAWMOCR35THP5CKCB6LZBW4GMJU6O6KKVSIJ3ALLONQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.668451284 (542h19m51.539534319s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.671076552 (542h19m51.542159587s) [noise=true , address=77.90.31.11:4560 , piece_id=HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.674568571 (542h19m51.545651626s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:52.998088476 (542h19m52.869171521s) [upload=475588]: Next returned nil segment
Dec 28 05:48:52.998103846 (542h19m52.869186881s) [upload=475588]: Waiting for error group managing segments...
Dec 28 05:48:53.543654582 (542h19m53.414737627s) [noise=true , address=145.79.227.40:4067 , piece_id=36Q7DHHN5ELVD3BYSNUGJ7WYXNGIDKERTOPRAZJNJ7YQTOUPWEYQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.543744133 (542h19m53.414827178s) [noise=true , address=145.79.227.49:4370 , piece_id=QDMDBC3576X7ZYGUAC2DLJKWBNVOLSROCMA4UNFDZ47TTADILUAQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.556254274 (542h19m53.427337309s) [noise=true , address=145.79.227.34:4915 , piece_id=MR5WOODFPXH57I6O2JCMC3EE5OWXF23CFLLATZXFP5XK6YPWDZJA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.810781113 (542h19m53.681864158s) [noise=true , address=77.90.31.13:4647 , piece_id=SOPU7ZUZQXRLU6TUL4L3ODX7U7WFAKUDOSK5UO66WMAIANU6PVUA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.814762255 (542h19m53.6858453s) [noise=true , address=77.90.31.16:4756 , piece_id=GY4LXIIIB7SVWA2WRAEHJVB6HLN6XQYISF7ZPVE4KA42C23MEQ4A, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.816966662 (542h19m53.688049707s) [noise=true , address=77.90.31.15:4727 , piece_id=3ACH3RUAR5KUXZU23AL3OQSGIZQOU5MFOFRN3JUYLQJBJCA4JOZQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.895077718 (542h19m53.766160783s) [noise=true , address=45.91.116.9:4290 , piece_id=MP5DAA5UGHWQ3BA5R7GBWNIV2B4JYCL5WUSV2PY7GTO4PNLH2F2Q, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.906586933 (542h19m53.777669978s) [noise=true , address=45.91.116.8:4059 , piece_id=OSA2YK4AH2DPQBNSJDFT3PLTGPJM6N7EOB7X27ZP2WM52HWSHT4Q, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.930785842 (542h19m53.801868877s) [noise=true , address=145.79.227.33:4876 , piece_id=E5QQDATADJON5NOOQ6X37GFKUQCFUKJRGK6G2EURBSUJVLF72Y7A, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.087446700 (542h19m53.958529735s) [noise=true , address=45.91.116.6:4998 , piece_id=FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.129981903 (542h19m54.001064938s) [noise=true , address=77.90.31.11:4560 , piece_id=HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.264740966 (542h19m54.135824011s) [noise=true , address=77.90.31.12:4594 , piece_id=DNXKJQOUHLAWMOCR35THP5CKCB6LZBW4GMJU6O6KKVSIJ3ALLONQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.368348401 (542h19m54.239431446s) [noise=true , address=77.90.31.18:4827 , piece_id=V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.437713885 (542h19m54.30879693s) [noise=true , address=145.79.227.44:4209 , piece_id=JRNDKXGHFY5OJYSY4ESURYMK5BNRDWY6INLHOWEK55KHMNRJO4AQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.437828132 (542h19m54.308911187s) [seg_pos={0 0}, upload=475588]: Segment reached optimal threshold of 14 pieces.
Dec 28 05:48:54.438064122 (542h19m54.309147167s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF
Dec 28 05:48:54.438074481 (542h19m54.309157546s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold
Dec 28 05:48:54.438099619 (542h19m54.309182664s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F4fJAGJM2ZPRt7vJH2cZmEbYeujmxcfi7aPZ2g9VPTNYBYRAi): protocol: expected piece hash; context canceled; EOF
Dec 28 05:48:54.438108586 (542h19m54.309191631s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold
Dec 28 05:48:54.438243403 (542h19m54.309326458s) [noise=true , address=77.90.31.19:4876 , piece_id=PUIQX7IZSNPKZTZTYH32KTFUM7K53KWZASLPT6JHAPMOB2JCXPGA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12H3UNBGeAufYSdVvEgYkv2QWeFtjKMvHvYGeqGeBvjEHqDDjBB): protocol: expected piece hash; context canceled; EOF
Dec 28 05:48:54.438252169 (542h19m54.309335214s) [noise=true , address=77.90.31.19:4876 , piece_id=PUIQX7IZSNPKZTZTYH32KTFUM7K53KWZASLPT6JHAPMOB2JCXPGA, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold
Dec 28 05:48:54.438391024 (542h19m54.309474069s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection
Dec 28 05:48:54.438398588 (542h19m54.309481633s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold
Dec 28 05:48:54.438699081 (542h19m54.309782126s) [seg_pos={0 0}, upload=475588]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 05:48:54.438794753 (542h19m54.309877798s) [upload=475588]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 05:48:54.438801255 (542h19m54.30988429s) [upload=475588]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 05:48:54.728435695 (542h19m54.59951876s) [upload=475588]: Done uploading segments...
Dec 28 05:48:54.728457938 (542h19m54.599540983s) [upload=475588]: Upload finished. err: <nil>
Duration: 42.100228101s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
23 out of 60 is failed (threshold=6) [.....XXX....XXXX....XXX.X...X....X...XX.X..XXX....X..XX.X...] duration:
3.41889667s trace:
https://jaeger.dev.storj.io/trace/1fac56088bbc7204
FAIL (2025-12-28 05:55:16 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.418924042s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
24 out of 60 is failed (threshold=6) [X.....XXX....XXXX....XXX.X...X....X...XX.X..XXX....X..XX.X..] duration:
5.000945351s trace:
https://jaeger.dev.storj.io/trace/1fac56088bbc7204
FAIL (2025-12-28 05:54:59 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000968384s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
20 out of 60 is failed (threshold=3) [.X.XX.X.XX..XX.X...X.X.XX....X...X.X..X.....X........X...X..] duration:
6.431367899s trace:
https://jaeger.dev.storj.io/trace/1b676c3e614b0838
FAIL (2025-12-28 05:54:42 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
05:53:12.372154109 (542h24m12.243237154s) [upload=475651]: Starting upload
Dec 28 05:53:12.372235544 (542h24m12.243318579s) [upload=475651]: Done starting upload
Dec 28 05:53:12.372298123 (542h24m12.243381158s) [upload=475651]: Uploading segments...
Dec 28 05:53:12.372455743 (542h24m12.243538799s) [upload=475651]: Got next segment. Inline: false
Dec 28 05:53:12.372500789 (542h24m12.243583824s) [upload=475651]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 05:53:12.372507171 (542h24m12.243590216s) [upload=475651]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 05:53:12.624874469 (542h24m12.495957514s) [seg_pos={0 0}, upload=475651]: Begin upload segment...
Dec 28 05:53:12.624935255 (542h24m12.49601832s) [seg_pos={0 0}, upload=475651]: Done begin upload segment.
Dec 28 05:53:12.624966685 (542h24m12.49604973s) [noise=true , address=77.90.31.14:4663 , piece_id=H42L5Y6KTBKTY6ALEF3N6IPVRYVQ7FUIDUUIMXHEXCY2GUMN5DCA, seg_pos={0 0}, upload=475651]: Uploading piece...
Dec 28 05:53:12.625162367 (542h24m12.496245412s) [noise=true , address=145.79.227.23:4547 , piece_id=VA36TDAORS4LZXQRRJGQTSM3OLWACA3U2GRSLRRA6YOUSUFPQPRQ, seg_pos={0 0}, upload=475651]: Uploading piece...
Dec 28 05:53:12.626802931 (542h24m12.497885976s) [noise=true , address=145.79.227.51:4411 , piece_id=S5WWPLTIF4CZNWXC3HHZAUQMKKTK7I5IBZJVAPGIEEWY7JPNVVFA, seg_pos={0 0}, upload=475651]: Uploading piece...
Dec 28 05:53:12.627534754 (542h24m12.498617809s) [noise=true , address=45.91.116.10:4083 , piece_id=ONIM7G4E2XBEFKOE3K23Z5GPVI7L4QXSI2DFHAPCI3SI7SEWGJXQ, seg_pos={0 0}, upload=475651]: Uploading piece...
Dec 28 05:53:12.628181346 (542h24m12.499264391s) [noise=true , address=145.79.227.14:4840 , piece_id=AKZWP4AKS5S45GQAAXJYSJNYP5LRWC2T7V63JSJJJUUJVJC2X6BA, seg_pos={0 0}, upload=475651]: Uploading piece...
Dec 28 05:53:12.628776179 (542h24m12.499859224s) [noise=true , address=145.79.227.8:4331 , piece_id=2BE2QILPDMMCTYABRMZ4NY66WCTKD5BEPUUQV2K6AWR6FZUFD6OQ, seg_pos={0 0}, upload=475651]: Uploading piece...
Dec 28 05:53:12.629280218 (542h24m12.500363263s) [noise=true , address=45.91.116.7:4009 , piece_id=LFCYGY7WKDBMYZZG4Q243SVZUOWXG7WP7ILOMA2LFKB423VQK6DA, seg_pos={0 0}, upload=475651]: Uploading piece...
Dec 28 05:53:12.629910809 (542h24m12.500993854s) [noise=true , address=145.79.227.49:4338 , piece_id=A6D2FN2VDHOBM35UGI2WUWOIA7Z535VHKBCPBAS6GOWYT4UA7OMA, seg_pos={0 0}, upload=475651]: Uploading piece...
Dec 28 05:53:12.632122791 (542h24m12.503205836s) [noise=true , address=145.79.227.50:4375 , piece_id=RKKFLFRN2IBMXFAFS47P2QJHPG4O4K4QKHD3PUXUEYBGGIWVXPQA, seg_pos={0 0}, upload=475651]: Uploading piece...
Dec 28 05:53:12.634261974 (542h24m12.505345019s) [noise=true , address=45.91.116.5:4298 , piece_id=VWLP2563LGIXGJHNT6LSAADDT6L3U7L4FOZ2IXYXTYSTNGXABARQ, seg_pos={0 0}, upload=475651]: Uploading piece...
Dec 28 05:53:12.636067763 (542h24m12.507150818s) [noise=true , address=145.79.227.3:4150 , piece_id=2ENQRZTSILOQ7ETJ43Z6ZFA4QMP6QH2DNQ5Y3HX2WCHOOLFSZNAQ, seg_pos={0 0}, upload=475651]: Uploading piece...
Dec 28 05:53:12.637926802 (542h24m12.509009847s) [noise=true , address=45.91.116.8:4047 , piece_id=4ZZCFH5CZU4OCXJ7JIR5JQJ3WNILN2M2KBDSSV3J376GTL3WFHKA, seg_pos={0 0}, upload=475651]: Uploading piece...
Dec 28 05:53:12.638985489 (542h24m12.510068534s) [noise=true , address=145.79.227.46:4602 , piece_id=O6ECNFY2ZMVCUNTEP5NTKIHSBEBBU6PRLTV26LC375TES5MSYLTA, seg_pos={0 0}, upload=475651]: Uploading piece...
Dec 28 05:53:12.640512226 (542h24m12.511595271s) [noise=true , address=145.79.227.17:4972 , piece_id=JKWFT2QM6BJDSLZ23LA55XAIWGGN6PFZHDGQURJEMMFGNNA7D2RA, seg_pos={0 0}, upload=475651]: Uploading piece...
Dec 28 05:53:12.642698118 (542h24m12.513781153s) [noise=true , address=145.79.227.39:4014 , piece_id=SQN5T33XGIQU64IFXZAFG3M4R2XVOMYWPB3EZ6CVKIC6V5IJGK3A, seg_pos={0 0}, upload=475651]: Uploading piece...
Dec 28 05:53:12.646531357 (542h24m12.517614403s) [noise=true , address=145.79.227.6:4258 , piece_id=NWXU2XBLIJBK43VY7XNIJU5A5OCPXKVSL5H3LDA5COFRUKWFLXRA, seg_pos={0 0}, upload=475651]: Uploading piece...
Dec 28 05:53:14.260734519 (542h24m14.131817554s) [upload=475651]: Next returned nil segment
Dec 28 05:53:14.260748215 (542h24m14.13183126s) [upload=475651]: Waiting for error group managing segments...
Dec 28 05:53:14.632918707 (542h24m14.504001743s) [noise=true , address=145.79.227.8:4331 , piece_id=2BE2QILPDMMCTYABRMZ4NY66WCTKD5BEPUUQV2K6AWR6FZUFD6OQ, seg_pos={0 0}, upload=475651]: Done uploading piece. err: <nil>
Dec 28 05:53:14.642873483 (542h24m14.513956528s) [noise=true , address=145.79.227.23:4547 , piece_id=VA36TDAORS4LZXQRRJGQTSM3OLWACA3U2GRSLRRA6YOUSUFPQPRQ, seg_pos={0 0}, upload=475651]: Done uploading piece. err: <nil>
Dec 28 05:53:14.651452178 (542h24m14.522535213s) [noise=true , address=145.79.227.14:4840 , piece_id=AKZWP4AKS5S45GQAAXJYSJNYP5LRWC2T7V63JSJJJUUJVJC2X6BA, seg_pos={0 0}, upload=475651]: Done uploading piece. err: <nil>
Dec 28 05:53:14.659488360 (542h24m14.530571405s) [noise=true , address=145.79.227.51:4411 , piece_id=S5WWPLTIF4CZNWXC3HHZAUQMKKTK7I5IBZJVAPGIEEWY7JPNVVFA, seg_pos={0 0}, upload=475651]: Done uploading piece. err: <nil>
Dec 28 05:53:14.678215400 (542h24m14.549298445s) [noise=true , address=145.79.227.6:4258 , piece_id=NWXU2XBLIJBK43VY7XNIJU5A5OCPXKVSL5H3LDA5COFRUKWFLXRA, seg_pos={0 0}, upload=475651]: Done uploading piece. err: <nil>
Dec 28 05:53:14.683000712 (542h24m14.554083747s) [noise=true , address=145.79.227.39:4014 , piece_id=SQN5T33XGIQU64IFXZAFG3M4R2XVOMYWPB3EZ6CVKIC6V5IJGK3A, seg_pos={0 0}, upload=475651]: Done uploading piece. err: <nil>
Dec 28 05:53:14.690028574 (542h24m14.561111619s) [noise=true , address=145.79.227.46:4602 , piece_id=O6ECNFY2ZMVCUNTEP5NTKIHSBEBBU6PRLTV26LC375TES5MSYLTA, seg_pos={0 0}, upload=475651]: Done uploading piece. err: <nil>
Dec 28 05:53:14.742783919 (542h24m14.613866964s) [noise=true , address=145.79.227.49:4338 , piece_id=A6D2FN2VDHOBM35UGI2WUWOIA7Z535VHKBCPBAS6GOWYT4UA7OMA, seg_pos={0 0}, upload=475651]: Done uploading piece. err: <nil>
Dec 28 05:53:14.743312715 (542h24m14.614395761s) [noise=true , address=145.79.227.17:4972 , piece_id=JKWFT2QM6BJDSLZ23LA55XAIWGGN6PFZHDGQURJEMMFGNNA7D2RA, seg_pos={0 0}, upload=475651]: Done uploading piece. err: <nil>
Dec 28 05:53:14.757172958 (542h24m14.628255993s) [noise=true , address=145.79.227.3:4150 , piece_id=2ENQRZTSILOQ7ETJ43Z6ZFA4QMP6QH2DNQ5Y3HX2WCHOOLFSZNAQ, seg_pos={0 0}, upload=475651]: Done uploading piece. err: <nil>
Dec 28 05:53:14.873879385 (542h24m14.74496242s) [noise=true , address=45.91.116.7:4009 , piece_id=LFCYGY7WKDBMYZZG4Q243SVZUOWXG7WP7ILOMA2LFKB423VQK6DA, seg_pos={0 0}, upload=475651]: Done uploading piece. err: <nil>
Dec 28 05:53:14.882595904 (542h24m14.753678949s) [noise=true , address=45.91.116.8:4047 , piece_id=4ZZCFH5CZU4OCXJ7JIR5JQJ3WNILN2M2KBDSSV3J376GTL3WFHKA, seg_pos={0 0}, upload=475651]: Done uploading piece. err: <nil>
Dec 28 05:53:14.960588954 (542h24m14.831671999s) [noise=true , address=77.90.31.14:4663 , piece_id=H42L5Y6KTBKTY6ALEF3N6IPVRYVQ7FUIDUUIMXHEXCY2GUMN5DCA, seg_pos={0 0}, upload=475651]: Done uploading piece. err: <nil>
Dec 28 05:53:14.960613722 (542h24m14.831696767s) [seg_pos={0 0}, upload=475651]: Segment reached optimal threshold of 13 pieces.
Dec 28 05:53:14.960846444 (542h24m14.831929479s) [noise=true , address=45.91.116.10:4083 , piece_id=ONIM7G4E2XBEFKOE3K23Z5GPVI7L4QXSI2DFHAPCI3SI7SEWGJXQ, seg_pos={0 0}, upload=475651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:45832->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:45832->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45832->45.91.116.10:4083: use of closed network connection
Dec 28 05:53:14.960853889 (542h24m14.831936924s) [noise=true , address=45.91.116.10:4083 , piece_id=ONIM7G4E2XBEFKOE3K23Z5GPVI7L4QXSI2DFHAPCI3SI7SEWGJXQ, seg_pos={0 0}, upload=475651]: Long tail cancellation due to optimal threshold
Dec 28 05:53:14.961131438 (542h24m14.832214473s) [noise=true , address=45.91.116.5:4298 , piece_id=VWLP2563LGIXGJHNT6LSAADDT6L3U7L4FOZ2IXYXTYSTNGXABARQ, seg_pos={0 0}, upload=475651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF
Dec 28 05:53:14.961138621 (542h24m14.832221656s) [noise=true , address=45.91.116.5:4298 , piece_id=VWLP2563LGIXGJHNT6LSAADDT6L3U7L4FOZ2IXYXTYSTNGXABARQ, seg_pos={0 0}, upload=475651]: Long tail cancellation due to optimal threshold
Dec 28 05:53:14.961175221 (542h24m14.832258266s) [noise=true , address=145.79.227.50:4375 , piece_id=RKKFLFRN2IBMXFAFS47P2QJHPG4O4K4QKHD3PUXUEYBGGIWVXPQA, seg_pos={0 0}, upload=475651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5): protocol: expected piece hash; context canceled; EOF
Dec 28 05:53:14.961207753 (542h24m14.832290818s) [noise=true , address=145.79.227.50:4375 , piece_id=RKKFLFRN2IBMXFAFS47P2QJHPG4O4K4QKHD3PUXUEYBGGIWVXPQA, seg_pos={0 0}, upload=475651]: Long tail cancellation due to optimal threshold
Dec 28 05:53:14.961235105 (542h24m14.83231815s) [seg_pos={0 0}, upload=475651]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 05:53:14.961335236 (542h24m14.832418301s) [upload=475651]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 05:53:14.961341397 (542h24m14.832424443s) [upload=475651]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 05:53:15.262227303 (542h24m15.133310348s) [upload=475651]: Done uploading segments...
Dec 28 05:53:15.262254815 (542h24m15.13333786s) [upload=475651]: Upload finished. err: <nil>
Duration: 6.43140047s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
25 out of 60 is failed (threshold=6) [XX.....XXX....XXXX....XXX.X...X....X...XX.X..XXX....X..XX.X.] duration:
4.195615637s trace:
https://jaeger.dev.storj.io/trace/1fac56088bbc7204
FAIL (2025-12-28 05:54:30 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.195645114s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [...................X.........X...X....................X.....] duration:
41.623804248s trace:
https://jaeger.dev.storj.io/trace/35a88c4b3a3d78b6
FAIL (2025-12-28 05:54:26 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH, Piece ID: HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA) context deadline exceeded; piecestore: (Node ID: 1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, Piece ID: GY4LXIIIB7SVWA2WRAEHJVB6HLN6XQYISF7ZPVE4KA42C23MEQ4A) context deadline exceeded; piecestore: (Node ID: 1RQ3vtArvKAvaMJTVa9VDDVdWR5oBuZ7gaPB3pzc1MouqBZhWu, Piece ID: FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ) context deadline exceeded; piecestore: (Node ID: 127KQ2X6VXddEeFRMau7MVGwtTsUmFsm8hTYDR2A6B4A1oVKfyo, Piece ID: V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ) context deadline exceeded
======LOGS======
) [noise=true , address=145.79.227.44:4209 , piece_id=JRNDKXGHFY5OJYSY4ESURYMK5BNRDWY6INLHOWEK55KHMNRJO4AQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.655023976 (542h19m51.526107021s) [noise=true , address=77.90.31.13:4647 , piece_id=SOPU7ZUZQXRLU6TUL4L3ODX7U7WFAKUDOSK5UO66WMAIANU6PVUA, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.655676448 (542h19m51.526759493s) [noise=true , address=45.91.116.8:4059 , piece_id=OSA2YK4AH2DPQBNSJDFT3PLTGPJM6N7EOB7X27ZP2WM52HWSHT4Q, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.656204463 (542h19m51.527287509s) [noise=true , address=145.79.227.49:4370 , piece_id=QDMDBC3576X7ZYGUAC2DLJKWBNVOLSROCMA4UNFDZ47TTADILUAQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.656699927 (542h19m51.527782972s) [noise=true , address=45.91.116.9:4290 , piece_id=MP5DAA5UGHWQ3BA5R7GBWNIV2B4JYCL5WUSV2PY7GTO4PNLH2F2Q, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.657183808 (542h19m51.528266883s) [noise=true , address=77.90.31.18:4827 , piece_id=V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.657682177 (542h19m51.528765222s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.658706768 (542h19m51.529789813s) [noise=true , address=145.79.227.33:4876 , piece_id=E5QQDATADJON5NOOQ6X37GFKUQCFUKJRGK6G2EURBSUJVLF72Y7A, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.660113707 (542h19m51.531196752s) [noise=true , address=145.79.227.34:4915 , piece_id=MR5WOODFPXH57I6O2JCMC3EE5OWXF23CFLLATZXFP5XK6YPWDZJA, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.661356113 (542h19m51.532439158s) [noise=true , address=77.90.31.15:4727 , piece_id=3ACH3RUAR5KUXZU23AL3OQSGIZQOU5MFOFRN3JUYLQJBJCA4JOZQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.664431910 (542h19m51.535514955s) [noise=true , address=45.91.116.6:4998 , piece_id=FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.666377204 (542h19m51.537460249s) [noise=true , address=145.79.227.40:4067 , piece_id=36Q7DHHN5ELVD3BYSNUGJ7WYXNGIDKERTOPRAZJNJ7YQTOUPWEYQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.666754502 (542h19m51.537837548s) [noise=true , address=77.90.31.12:4594 , piece_id=DNXKJQOUHLAWMOCR35THP5CKCB6LZBW4GMJU6O6KKVSIJ3ALLONQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.668451284 (542h19m51.539534319s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.671076552 (542h19m51.542159587s) [noise=true , address=77.90.31.11:4560 , piece_id=HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.674568571 (542h19m51.545651626s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:52.998088476 (542h19m52.869171521s) [upload=475588]: Next returned nil segment
Dec 28 05:48:52.998103846 (542h19m52.869186881s) [upload=475588]: Waiting for error group managing segments...
Dec 28 05:48:53.543654582 (542h19m53.414737627s) [noise=true , address=145.79.227.40:4067 , piece_id=36Q7DHHN5ELVD3BYSNUGJ7WYXNGIDKERTOPRAZJNJ7YQTOUPWEYQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.543744133 (542h19m53.414827178s) [noise=true , address=145.79.227.49:4370 , piece_id=QDMDBC3576X7ZYGUAC2DLJKWBNVOLSROCMA4UNFDZ47TTADILUAQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.556254274 (542h19m53.427337309s) [noise=true , address=145.79.227.34:4915 , piece_id=MR5WOODFPXH57I6O2JCMC3EE5OWXF23CFLLATZXFP5XK6YPWDZJA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.810781113 (542h19m53.681864158s) [noise=true , address=77.90.31.13:4647 , piece_id=SOPU7ZUZQXRLU6TUL4L3ODX7U7WFAKUDOSK5UO66WMAIANU6PVUA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.814762255 (542h19m53.6858453s) [noise=true , address=77.90.31.16:4756 , piece_id=GY4LXIIIB7SVWA2WRAEHJVB6HLN6XQYISF7ZPVE4KA42C23MEQ4A, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.816966662 (542h19m53.688049707s) [noise=true , address=77.90.31.15:4727 , piece_id=3ACH3RUAR5KUXZU23AL3OQSGIZQOU5MFOFRN3JUYLQJBJCA4JOZQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.895077718 (542h19m53.766160783s) [noise=true , address=45.91.116.9:4290 , piece_id=MP5DAA5UGHWQ3BA5R7GBWNIV2B4JYCL5WUSV2PY7GTO4PNLH2F2Q, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.906586933 (542h19m53.777669978s) [noise=true , address=45.91.116.8:4059 , piece_id=OSA2YK4AH2DPQBNSJDFT3PLTGPJM6N7EOB7X27ZP2WM52HWSHT4Q, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.930785842 (542h19m53.801868877s) [noise=true , address=145.79.227.33:4876 , piece_id=E5QQDATADJON5NOOQ6X37GFKUQCFUKJRGK6G2EURBSUJVLF72Y7A, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.087446700 (542h19m53.958529735s) [noise=true , address=45.91.116.6:4998 , piece_id=FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.129981903 (542h19m54.001064938s) [noise=true , address=77.90.31.11:4560 , piece_id=HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.264740966 (542h19m54.135824011s) [noise=true , address=77.90.31.12:4594 , piece_id=DNXKJQOUHLAWMOCR35THP5CKCB6LZBW4GMJU6O6KKVSIJ3ALLONQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.368348401 (542h19m54.239431446s) [noise=true , address=77.90.31.18:4827 , piece_id=V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.437713885 (542h19m54.30879693s) [noise=true , address=145.79.227.44:4209 , piece_id=JRNDKXGHFY5OJYSY4ESURYMK5BNRDWY6INLHOWEK55KHMNRJO4AQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.437828132 (542h19m54.308911187s) [seg_pos={0 0}, upload=475588]: Segment reached optimal threshold of 14 pieces.
Dec 28 05:48:54.438064122 (542h19m54.309147167s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF
Dec 28 05:48:54.438074481 (542h19m54.309157546s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold
Dec 28 05:48:54.438099619 (542h19m54.309182664s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F4fJAGJM2ZPRt7vJH2cZmEbYeujmxcfi7aPZ2g9VPTNYBYRAi): protocol: expected piece hash; context canceled; EOF
Dec 28 05:48:54.438108586 (542h19m54.309191631s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold
Dec 28 05:48:54.438243403 (542h19m54.309326458s) [noise=true , address=77.90.31.19:4876 , piece_id=PUIQX7IZSNPKZTZTYH32KTFUM7K53KWZASLPT6JHAPMOB2JCXPGA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12H3UNBGeAufYSdVvEgYkv2QWeFtjKMvHvYGeqGeBvjEHqDDjBB): protocol: expected piece hash; context canceled; EOF
Dec 28 05:48:54.438252169 (542h19m54.309335214s) [noise=true , address=77.90.31.19:4876 , piece_id=PUIQX7IZSNPKZTZTYH32KTFUM7K53KWZASLPT6JHAPMOB2JCXPGA, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold
Dec 28 05:48:54.438391024 (542h19m54.309474069s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection
Dec 28 05:48:54.438398588 (542h19m54.309481633s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold
Dec 28 05:48:54.438699081 (542h19m54.309782126s) [seg_pos={0 0}, upload=475588]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 05:48:54.438794753 (542h19m54.309877798s) [upload=475588]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 05:48:54.438801255 (542h19m54.30988429s) [upload=475588]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 05:48:54.728435695 (542h19m54.59951876s) [upload=475588]: Done uploading segments...
Dec 28 05:48:54.728457938 (542h19m54.599540983s) [upload=475588]: Upload finished. err: <nil>
Duration: 41.623842299s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
20 out of 60 is failed (threshold=3) [..X.XX.X.XX..XX.X...X.X.XX....X...X.X..X.....X........X...X.] duration:
6.207256945s trace:
https://jaeger.dev.storj.io/trace/1b676c3e614b0838
FAIL (2025-12-28 05:54:02 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
05:53:12.372154109 (542h24m12.243237154s) [upload=475651]: Starting upload
Dec 28 05:53:12.372235544 (542h24m12.243318579s) [upload=475651]: Done starting upload
Dec 28 05:53:12.372298123 (542h24m12.243381158s) [upload=475651]: Uploading segments...
Dec 28 05:53:12.372455743 (542h24m12.243538799s) [upload=475651]: Got next segment. Inline: false
Dec 28 05:53:12.372500789 (542h24m12.243583824s) [upload=475651]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 05:53:12.372507171 (542h24m12.243590216s) [upload=475651]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 05:53:12.624874469 (542h24m12.495957514s) [seg_pos={0 0}, upload=475651]: Begin upload segment...
Dec 28 05:53:12.624935255 (542h24m12.49601832s) [seg_pos={0 0}, upload=475651]: Done begin upload segment.
Dec 28 05:53:12.624966685 (542h24m12.49604973s) [noise=true , address=77.90.31.14:4663 , piece_id=H42L5Y6KTBKTY6ALEF3N6IPVRYVQ7FUIDUUIMXHEXCY2GUMN5DCA, seg_pos={0 0}, upload=475651]: Uploading piece...
Dec 28 05:53:12.625162367 (542h24m12.496245412s) [noise=true , address=145.79.227.23:4547 , piece_id=VA36TDAORS4LZXQRRJGQTSM3OLWACA3U2GRSLRRA6YOUSUFPQPRQ, seg_pos={0 0}, upload=475651]: Uploading piece...
Dec 28 05:53:12.626802931 (542h24m12.497885976s) [noise=true , address=145.79.227.51:4411 , piece_id=S5WWPLTIF4CZNWXC3HHZAUQMKKTK7I5IBZJVAPGIEEWY7JPNVVFA, seg_pos={0 0}, upload=475651]: Uploading piece...
Dec 28 05:53:12.627534754 (542h24m12.498617809s) [noise=true , address=45.91.116.10:4083 , piece_id=ONIM7G4E2XBEFKOE3K23Z5GPVI7L4QXSI2DFHAPCI3SI7SEWGJXQ, seg_pos={0 0}, upload=475651]: Uploading piece...
Dec 28 05:53:12.628181346 (542h24m12.499264391s) [noise=true , address=145.79.227.14:4840 , piece_id=AKZWP4AKS5S45GQAAXJYSJNYP5LRWC2T7V63JSJJJUUJVJC2X6BA, seg_pos={0 0}, upload=475651]: Uploading piece...
Dec 28 05:53:12.628776179 (542h24m12.499859224s) [noise=true , address=145.79.227.8:4331 , piece_id=2BE2QILPDMMCTYABRMZ4NY66WCTKD5BEPUUQV2K6AWR6FZUFD6OQ, seg_pos={0 0}, upload=475651]: Uploading piece...
Dec 28 05:53:12.629280218 (542h24m12.500363263s) [noise=true , address=45.91.116.7:4009 , piece_id=LFCYGY7WKDBMYZZG4Q243SVZUOWXG7WP7ILOMA2LFKB423VQK6DA, seg_pos={0 0}, upload=475651]: Uploading piece...
Dec 28 05:53:12.629910809 (542h24m12.500993854s) [noise=true , address=145.79.227.49:4338 , piece_id=A6D2FN2VDHOBM35UGI2WUWOIA7Z535VHKBCPBAS6GOWYT4UA7OMA, seg_pos={0 0}, upload=475651]: Uploading piece...
Dec 28 05:53:12.632122791 (542h24m12.503205836s) [noise=true , address=145.79.227.50:4375 , piece_id=RKKFLFRN2IBMXFAFS47P2QJHPG4O4K4QKHD3PUXUEYBGGIWVXPQA, seg_pos={0 0}, upload=475651]: Uploading piece...
Dec 28 05:53:12.634261974 (542h24m12.505345019s) [noise=true , address=45.91.116.5:4298 , piece_id=VWLP2563LGIXGJHNT6LSAADDT6L3U7L4FOZ2IXYXTYSTNGXABARQ, seg_pos={0 0}, upload=475651]: Uploading piece...
Dec 28 05:53:12.636067763 (542h24m12.507150818s) [noise=true , address=145.79.227.3:4150 , piece_id=2ENQRZTSILOQ7ETJ43Z6ZFA4QMP6QH2DNQ5Y3HX2WCHOOLFSZNAQ, seg_pos={0 0}, upload=475651]: Uploading piece...
Dec 28 05:53:12.637926802 (542h24m12.509009847s) [noise=true , address=45.91.116.8:4047 , piece_id=4ZZCFH5CZU4OCXJ7JIR5JQJ3WNILN2M2KBDSSV3J376GTL3WFHKA, seg_pos={0 0}, upload=475651]: Uploading piece...
Dec 28 05:53:12.638985489 (542h24m12.510068534s) [noise=true , address=145.79.227.46:4602 , piece_id=O6ECNFY2ZMVCUNTEP5NTKIHSBEBBU6PRLTV26LC375TES5MSYLTA, seg_pos={0 0}, upload=475651]: Uploading piece...
Dec 28 05:53:12.640512226 (542h24m12.511595271s) [noise=true , address=145.79.227.17:4972 , piece_id=JKWFT2QM6BJDSLZ23LA55XAIWGGN6PFZHDGQURJEMMFGNNA7D2RA, seg_pos={0 0}, upload=475651]: Uploading piece...
Dec 28 05:53:12.642698118 (542h24m12.513781153s) [noise=true , address=145.79.227.39:4014 , piece_id=SQN5T33XGIQU64IFXZAFG3M4R2XVOMYWPB3EZ6CVKIC6V5IJGK3A, seg_pos={0 0}, upload=475651]: Uploading piece...
Dec 28 05:53:12.646531357 (542h24m12.517614403s) [noise=true , address=145.79.227.6:4258 , piece_id=NWXU2XBLIJBK43VY7XNIJU5A5OCPXKVSL5H3LDA5COFRUKWFLXRA, seg_pos={0 0}, upload=475651]: Uploading piece...
Dec 28 05:53:14.260734519 (542h24m14.131817554s) [upload=475651]: Next returned nil segment
Dec 28 05:53:14.260748215 (542h24m14.13183126s) [upload=475651]: Waiting for error group managing segments...
Dec 28 05:53:14.632918707 (542h24m14.504001743s) [noise=true , address=145.79.227.8:4331 , piece_id=2BE2QILPDMMCTYABRMZ4NY66WCTKD5BEPUUQV2K6AWR6FZUFD6OQ, seg_pos={0 0}, upload=475651]: Done uploading piece. err: <nil>
Dec 28 05:53:14.642873483 (542h24m14.513956528s) [noise=true , address=145.79.227.23:4547 , piece_id=VA36TDAORS4LZXQRRJGQTSM3OLWACA3U2GRSLRRA6YOUSUFPQPRQ, seg_pos={0 0}, upload=475651]: Done uploading piece. err: <nil>
Dec 28 05:53:14.651452178 (542h24m14.522535213s) [noise=true , address=145.79.227.14:4840 , piece_id=AKZWP4AKS5S45GQAAXJYSJNYP5LRWC2T7V63JSJJJUUJVJC2X6BA, seg_pos={0 0}, upload=475651]: Done uploading piece. err: <nil>
Dec 28 05:53:14.659488360 (542h24m14.530571405s) [noise=true , address=145.79.227.51:4411 , piece_id=S5WWPLTIF4CZNWXC3HHZAUQMKKTK7I5IBZJVAPGIEEWY7JPNVVFA, seg_pos={0 0}, upload=475651]: Done uploading piece. err: <nil>
Dec 28 05:53:14.678215400 (542h24m14.549298445s) [noise=true , address=145.79.227.6:4258 , piece_id=NWXU2XBLIJBK43VY7XNIJU5A5OCPXKVSL5H3LDA5COFRUKWFLXRA, seg_pos={0 0}, upload=475651]: Done uploading piece. err: <nil>
Dec 28 05:53:14.683000712 (542h24m14.554083747s) [noise=true , address=145.79.227.39:4014 , piece_id=SQN5T33XGIQU64IFXZAFG3M4R2XVOMYWPB3EZ6CVKIC6V5IJGK3A, seg_pos={0 0}, upload=475651]: Done uploading piece. err: <nil>
Dec 28 05:53:14.690028574 (542h24m14.561111619s) [noise=true , address=145.79.227.46:4602 , piece_id=O6ECNFY2ZMVCUNTEP5NTKIHSBEBBU6PRLTV26LC375TES5MSYLTA, seg_pos={0 0}, upload=475651]: Done uploading piece. err: <nil>
Dec 28 05:53:14.742783919 (542h24m14.613866964s) [noise=true , address=145.79.227.49:4338 , piece_id=A6D2FN2VDHOBM35UGI2WUWOIA7Z535VHKBCPBAS6GOWYT4UA7OMA, seg_pos={0 0}, upload=475651]: Done uploading piece. err: <nil>
Dec 28 05:53:14.743312715 (542h24m14.614395761s) [noise=true , address=145.79.227.17:4972 , piece_id=JKWFT2QM6BJDSLZ23LA55XAIWGGN6PFZHDGQURJEMMFGNNA7D2RA, seg_pos={0 0}, upload=475651]: Done uploading piece. err: <nil>
Dec 28 05:53:14.757172958 (542h24m14.628255993s) [noise=true , address=145.79.227.3:4150 , piece_id=2ENQRZTSILOQ7ETJ43Z6ZFA4QMP6QH2DNQ5Y3HX2WCHOOLFSZNAQ, seg_pos={0 0}, upload=475651]: Done uploading piece. err: <nil>
Dec 28 05:53:14.873879385 (542h24m14.74496242s) [noise=true , address=45.91.116.7:4009 , piece_id=LFCYGY7WKDBMYZZG4Q243SVZUOWXG7WP7ILOMA2LFKB423VQK6DA, seg_pos={0 0}, upload=475651]: Done uploading piece. err: <nil>
Dec 28 05:53:14.882595904 (542h24m14.753678949s) [noise=true , address=45.91.116.8:4047 , piece_id=4ZZCFH5CZU4OCXJ7JIR5JQJ3WNILN2M2KBDSSV3J376GTL3WFHKA, seg_pos={0 0}, upload=475651]: Done uploading piece. err: <nil>
Dec 28 05:53:14.960588954 (542h24m14.831671999s) [noise=true , address=77.90.31.14:4663 , piece_id=H42L5Y6KTBKTY6ALEF3N6IPVRYVQ7FUIDUUIMXHEXCY2GUMN5DCA, seg_pos={0 0}, upload=475651]: Done uploading piece. err: <nil>
Dec 28 05:53:14.960613722 (542h24m14.831696767s) [seg_pos={0 0}, upload=475651]: Segment reached optimal threshold of 13 pieces.
Dec 28 05:53:14.960846444 (542h24m14.831929479s) [noise=true , address=45.91.116.10:4083 , piece_id=ONIM7G4E2XBEFKOE3K23Z5GPVI7L4QXSI2DFHAPCI3SI7SEWGJXQ, seg_pos={0 0}, upload=475651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:45832->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:45832->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45832->45.91.116.10:4083: use of closed network connection
Dec 28 05:53:14.960853889 (542h24m14.831936924s) [noise=true , address=45.91.116.10:4083 , piece_id=ONIM7G4E2XBEFKOE3K23Z5GPVI7L4QXSI2DFHAPCI3SI7SEWGJXQ, seg_pos={0 0}, upload=475651]: Long tail cancellation due to optimal threshold
Dec 28 05:53:14.961131438 (542h24m14.832214473s) [noise=true , address=45.91.116.5:4298 , piece_id=VWLP2563LGIXGJHNT6LSAADDT6L3U7L4FOZ2IXYXTYSTNGXABARQ, seg_pos={0 0}, upload=475651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF
Dec 28 05:53:14.961138621 (542h24m14.832221656s) [noise=true , address=45.91.116.5:4298 , piece_id=VWLP2563LGIXGJHNT6LSAADDT6L3U7L4FOZ2IXYXTYSTNGXABARQ, seg_pos={0 0}, upload=475651]: Long tail cancellation due to optimal threshold
Dec 28 05:53:14.961175221 (542h24m14.832258266s) [noise=true , address=145.79.227.50:4375 , piece_id=RKKFLFRN2IBMXFAFS47P2QJHPG4O4K4QKHD3PUXUEYBGGIWVXPQA, seg_pos={0 0}, upload=475651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5): protocol: expected piece hash; context canceled; EOF
Dec 28 05:53:14.961207753 (542h24m14.832290818s) [noise=true , address=145.79.227.50:4375 , piece_id=RKKFLFRN2IBMXFAFS47P2QJHPG4O4K4QKHD3PUXUEYBGGIWVXPQA, seg_pos={0 0}, upload=475651]: Long tail cancellation due to optimal threshold
Dec 28 05:53:14.961235105 (542h24m14.83231815s) [seg_pos={0 0}, upload=475651]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 05:53:14.961335236 (542h24m14.832418301s) [upload=475651]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 05:53:14.961341397 (542h24m14.832424443s) [upload=475651]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 05:53:15.262227303 (542h24m15.133310348s) [upload=475651]: Done uploading segments...
Dec 28 05:53:15.262254815 (542h24m15.13333786s) [upload=475651]: Upload finished. err: <nil>
Duration: 6.207283696s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
20 out of 60 is failed (threshold=3) [...X.XX.X.XX..XX.X...X.X.XX....X...X.X..X.....X........X...X] duration:
30.002573188s trace:
https://jaeger.dev.storj.io/trace/1b676c3e614b0838
FAIL (2025-12-28 05:53:41 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
05:53:12.372154109 (542h24m12.243237154s) [upload=475651]: Starting upload
Dec 28 05:53:12.372235544 (542h24m12.243318579s) [upload=475651]: Done starting upload
Dec 28 05:53:12.372298123 (542h24m12.243381158s) [upload=475651]: Uploading segments...
Dec 28 05:53:12.372455743 (542h24m12.243538799s) [upload=475651]: Got next segment. Inline: false
Dec 28 05:53:12.372500789 (542h24m12.243583824s) [upload=475651]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 05:53:12.372507171 (542h24m12.243590216s) [upload=475651]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 05:53:12.624874469 (542h24m12.495957514s) [seg_pos={0 0}, upload=475651]: Begin upload segment...
Dec 28 05:53:12.624935255 (542h24m12.49601832s) [seg_pos={0 0}, upload=475651]: Done begin upload segment.
Dec 28 05:53:12.624966685 (542h24m12.49604973s) [noise=true , address=77.90.31.14:4663 , piece_id=H42L5Y6KTBKTY6ALEF3N6IPVRYVQ7FUIDUUIMXHEXCY2GUMN5DCA, seg_pos={0 0}, upload=475651]: Uploading piece...
Dec 28 05:53:12.625162367 (542h24m12.496245412s) [noise=true , address=145.79.227.23:4547 , piece_id=VA36TDAORS4LZXQRRJGQTSM3OLWACA3U2GRSLRRA6YOUSUFPQPRQ, seg_pos={0 0}, upload=475651]: Uploading piece...
Dec 28 05:53:12.626802931 (542h24m12.497885976s) [noise=true , address=145.79.227.51:4411 , piece_id=S5WWPLTIF4CZNWXC3HHZAUQMKKTK7I5IBZJVAPGIEEWY7JPNVVFA, seg_pos={0 0}, upload=475651]: Uploading piece...
Dec 28 05:53:12.627534754 (542h24m12.498617809s) [noise=true , address=45.91.116.10:4083 , piece_id=ONIM7G4E2XBEFKOE3K23Z5GPVI7L4QXSI2DFHAPCI3SI7SEWGJXQ, seg_pos={0 0}, upload=475651]: Uploading piece...
Dec 28 05:53:12.628181346 (542h24m12.499264391s) [noise=true , address=145.79.227.14:4840 , piece_id=AKZWP4AKS5S45GQAAXJYSJNYP5LRWC2T7V63JSJJJUUJVJC2X6BA, seg_pos={0 0}, upload=475651]: Uploading piece...
Dec 28 05:53:12.628776179 (542h24m12.499859224s) [noise=true , address=145.79.227.8:4331 , piece_id=2BE2QILPDMMCTYABRMZ4NY66WCTKD5BEPUUQV2K6AWR6FZUFD6OQ, seg_pos={0 0}, upload=475651]: Uploading piece...
Dec 28 05:53:12.629280218 (542h24m12.500363263s) [noise=true , address=45.91.116.7:4009 , piece_id=LFCYGY7WKDBMYZZG4Q243SVZUOWXG7WP7ILOMA2LFKB423VQK6DA, seg_pos={0 0}, upload=475651]: Uploading piece...
Dec 28 05:53:12.629910809 (542h24m12.500993854s) [noise=true , address=145.79.227.49:4338 , piece_id=A6D2FN2VDHOBM35UGI2WUWOIA7Z535VHKBCPBAS6GOWYT4UA7OMA, seg_pos={0 0}, upload=475651]: Uploading piece...
Dec 28 05:53:12.632122791 (542h24m12.503205836s) [noise=true , address=145.79.227.50:4375 , piece_id=RKKFLFRN2IBMXFAFS47P2QJHPG4O4K4QKHD3PUXUEYBGGIWVXPQA, seg_pos={0 0}, upload=475651]: Uploading piece...
Dec 28 05:53:12.634261974 (542h24m12.505345019s) [noise=true , address=45.91.116.5:4298 , piece_id=VWLP2563LGIXGJHNT6LSAADDT6L3U7L4FOZ2IXYXTYSTNGXABARQ, seg_pos={0 0}, upload=475651]: Uploading piece...
Dec 28 05:53:12.636067763 (542h24m12.507150818s) [noise=true , address=145.79.227.3:4150 , piece_id=2ENQRZTSILOQ7ETJ43Z6ZFA4QMP6QH2DNQ5Y3HX2WCHOOLFSZNAQ, seg_pos={0 0}, upload=475651]: Uploading piece...
Dec 28 05:53:12.637926802 (542h24m12.509009847s) [noise=true , address=45.91.116.8:4047 , piece_id=4ZZCFH5CZU4OCXJ7JIR5JQJ3WNILN2M2KBDSSV3J376GTL3WFHKA, seg_pos={0 0}, upload=475651]: Uploading piece...
Dec 28 05:53:12.638985489 (542h24m12.510068534s) [noise=true , address=145.79.227.46:4602 , piece_id=O6ECNFY2ZMVCUNTEP5NTKIHSBEBBU6PRLTV26LC375TES5MSYLTA, seg_pos={0 0}, upload=475651]: Uploading piece...
Dec 28 05:53:12.640512226 (542h24m12.511595271s) [noise=true , address=145.79.227.17:4972 , piece_id=JKWFT2QM6BJDSLZ23LA55XAIWGGN6PFZHDGQURJEMMFGNNA7D2RA, seg_pos={0 0}, upload=475651]: Uploading piece...
Dec 28 05:53:12.642698118 (542h24m12.513781153s) [noise=true , address=145.79.227.39:4014 , piece_id=SQN5T33XGIQU64IFXZAFG3M4R2XVOMYWPB3EZ6CVKIC6V5IJGK3A, seg_pos={0 0}, upload=475651]: Uploading piece...
Dec 28 05:53:12.646531357 (542h24m12.517614403s) [noise=true , address=145.79.227.6:4258 , piece_id=NWXU2XBLIJBK43VY7XNIJU5A5OCPXKVSL5H3LDA5COFRUKWFLXRA, seg_pos={0 0}, upload=475651]: Uploading piece...
Dec 28 05:53:14.260734519 (542h24m14.131817554s) [upload=475651]: Next returned nil segment
Dec 28 05:53:14.260748215 (542h24m14.13183126s) [upload=475651]: Waiting for error group managing segments...
Dec 28 05:53:14.632918707 (542h24m14.504001743s) [noise=true , address=145.79.227.8:4331 , piece_id=2BE2QILPDMMCTYABRMZ4NY66WCTKD5BEPUUQV2K6AWR6FZUFD6OQ, seg_pos={0 0}, upload=475651]: Done uploading piece. err: <nil>
Dec 28 05:53:14.642873483 (542h24m14.513956528s) [noise=true , address=145.79.227.23:4547 , piece_id=VA36TDAORS4LZXQRRJGQTSM3OLWACA3U2GRSLRRA6YOUSUFPQPRQ, seg_pos={0 0}, upload=475651]: Done uploading piece. err: <nil>
Dec 28 05:53:14.651452178 (542h24m14.522535213s) [noise=true , address=145.79.227.14:4840 , piece_id=AKZWP4AKS5S45GQAAXJYSJNYP5LRWC2T7V63JSJJJUUJVJC2X6BA, seg_pos={0 0}, upload=475651]: Done uploading piece. err: <nil>
Dec 28 05:53:14.659488360 (542h24m14.530571405s) [noise=true , address=145.79.227.51:4411 , piece_id=S5WWPLTIF4CZNWXC3HHZAUQMKKTK7I5IBZJVAPGIEEWY7JPNVVFA, seg_pos={0 0}, upload=475651]: Done uploading piece. err: <nil>
Dec 28 05:53:14.678215400 (542h24m14.549298445s) [noise=true , address=145.79.227.6:4258 , piece_id=NWXU2XBLIJBK43VY7XNIJU5A5OCPXKVSL5H3LDA5COFRUKWFLXRA, seg_pos={0 0}, upload=475651]: Done uploading piece. err: <nil>
Dec 28 05:53:14.683000712 (542h24m14.554083747s) [noise=true , address=145.79.227.39:4014 , piece_id=SQN5T33XGIQU64IFXZAFG3M4R2XVOMYWPB3EZ6CVKIC6V5IJGK3A, seg_pos={0 0}, upload=475651]: Done uploading piece. err: <nil>
Dec 28 05:53:14.690028574 (542h24m14.561111619s) [noise=true , address=145.79.227.46:4602 , piece_id=O6ECNFY2ZMVCUNTEP5NTKIHSBEBBU6PRLTV26LC375TES5MSYLTA, seg_pos={0 0}, upload=475651]: Done uploading piece. err: <nil>
Dec 28 05:53:14.742783919 (542h24m14.613866964s) [noise=true , address=145.79.227.49:4338 , piece_id=A6D2FN2VDHOBM35UGI2WUWOIA7Z535VHKBCPBAS6GOWYT4UA7OMA, seg_pos={0 0}, upload=475651]: Done uploading piece. err: <nil>
Dec 28 05:53:14.743312715 (542h24m14.614395761s) [noise=true , address=145.79.227.17:4972 , piece_id=JKWFT2QM6BJDSLZ23LA55XAIWGGN6PFZHDGQURJEMMFGNNA7D2RA, seg_pos={0 0}, upload=475651]: Done uploading piece. err: <nil>
Dec 28 05:53:14.757172958 (542h24m14.628255993s) [noise=true , address=145.79.227.3:4150 , piece_id=2ENQRZTSILOQ7ETJ43Z6ZFA4QMP6QH2DNQ5Y3HX2WCHOOLFSZNAQ, seg_pos={0 0}, upload=475651]: Done uploading piece. err: <nil>
Dec 28 05:53:14.873879385 (542h24m14.74496242s) [noise=true , address=45.91.116.7:4009 , piece_id=LFCYGY7WKDBMYZZG4Q243SVZUOWXG7WP7ILOMA2LFKB423VQK6DA, seg_pos={0 0}, upload=475651]: Done uploading piece. err: <nil>
Dec 28 05:53:14.882595904 (542h24m14.753678949s) [noise=true , address=45.91.116.8:4047 , piece_id=4ZZCFH5CZU4OCXJ7JIR5JQJ3WNILN2M2KBDSSV3J376GTL3WFHKA, seg_pos={0 0}, upload=475651]: Done uploading piece. err: <nil>
Dec 28 05:53:14.960588954 (542h24m14.831671999s) [noise=true , address=77.90.31.14:4663 , piece_id=H42L5Y6KTBKTY6ALEF3N6IPVRYVQ7FUIDUUIMXHEXCY2GUMN5DCA, seg_pos={0 0}, upload=475651]: Done uploading piece. err: <nil>
Dec 28 05:53:14.960613722 (542h24m14.831696767s) [seg_pos={0 0}, upload=475651]: Segment reached optimal threshold of 13 pieces.
Dec 28 05:53:14.960846444 (542h24m14.831929479s) [noise=true , address=45.91.116.10:4083 , piece_id=ONIM7G4E2XBEFKOE3K23Z5GPVI7L4QXSI2DFHAPCI3SI7SEWGJXQ, seg_pos={0 0}, upload=475651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: write tcp 188.166.225.170:45832->45.91.116.10:4083: use of closed network connection; write tcp 188.166.225.170:45832->45.91.116.10:4083: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45832->45.91.116.10:4083: use of closed network connection
Dec 28 05:53:14.960853889 (542h24m14.831936924s) [noise=true , address=45.91.116.10:4083 , piece_id=ONIM7G4E2XBEFKOE3K23Z5GPVI7L4QXSI2DFHAPCI3SI7SEWGJXQ, seg_pos={0 0}, upload=475651]: Long tail cancellation due to optimal threshold
Dec 28 05:53:14.961131438 (542h24m14.832214473s) [noise=true , address=45.91.116.5:4298 , piece_id=VWLP2563LGIXGJHNT6LSAADDT6L3U7L4FOZ2IXYXTYSTNGXABARQ, seg_pos={0 0}, upload=475651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF
Dec 28 05:53:14.961138621 (542h24m14.832221656s) [noise=true , address=45.91.116.5:4298 , piece_id=VWLP2563LGIXGJHNT6LSAADDT6L3U7L4FOZ2IXYXTYSTNGXABARQ, seg_pos={0 0}, upload=475651]: Long tail cancellation due to optimal threshold
Dec 28 05:53:14.961175221 (542h24m14.832258266s) [noise=true , address=145.79.227.50:4375 , piece_id=RKKFLFRN2IBMXFAFS47P2QJHPG4O4K4QKHD3PUXUEYBGGIWVXPQA, seg_pos={0 0}, upload=475651]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121fzX9v4rgEd78xnDAwcQtthyQHV3yUZUQYLXZ9byPDucGK4T5): protocol: expected piece hash; context canceled; EOF
Dec 28 05:53:14.961207753 (542h24m14.832290818s) [noise=true , address=145.79.227.50:4375 , piece_id=RKKFLFRN2IBMXFAFS47P2QJHPG4O4K4QKHD3PUXUEYBGGIWVXPQA, seg_pos={0 0}, upload=475651]: Long tail cancellation due to optimal threshold
Dec 28 05:53:14.961235105 (542h24m14.83231815s) [seg_pos={0 0}, upload=475651]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 05:53:14.961335236 (542h24m14.832418301s) [upload=475651]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 05:53:14.961341397 (542h24m14.832424443s) [upload=475651]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 05:53:15.262227303 (542h24m15.133310348s) [upload=475651]: Done uploading segments...
Dec 28 05:53:15.262254815 (542h24m15.13333786s) [upload=475651]: Upload finished. err: <nil>
Duration: 30.002599498s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [....................X.........X...X....................X....] duration:
48.722085954s trace:
https://jaeger.dev.storj.io/trace/35a88c4b3a3d78b6
FAIL (2025-12-28 05:53:32 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH, Piece ID: HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA) context deadline exceeded; piecestore: (Node ID: 1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, Piece ID: GY4LXIIIB7SVWA2WRAEHJVB6HLN6XQYISF7ZPVE4KA42C23MEQ4A) context deadline exceeded; piecestore: (Node ID: 1RQ3vtArvKAvaMJTVa9VDDVdWR5oBuZ7gaPB3pzc1MouqBZhWu, Piece ID: FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ) context deadline exceeded; piecestore: (Node ID: 127KQ2X6VXddEeFRMau7MVGwtTsUmFsm8hTYDR2A6B4A1oVKfyo, Piece ID: V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ) context deadline exceeded
======LOGS======
) [noise=true , address=145.79.227.44:4209 , piece_id=JRNDKXGHFY5OJYSY4ESURYMK5BNRDWY6INLHOWEK55KHMNRJO4AQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.655023976 (542h19m51.526107021s) [noise=true , address=77.90.31.13:4647 , piece_id=SOPU7ZUZQXRLU6TUL4L3ODX7U7WFAKUDOSK5UO66WMAIANU6PVUA, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.655676448 (542h19m51.526759493s) [noise=true , address=45.91.116.8:4059 , piece_id=OSA2YK4AH2DPQBNSJDFT3PLTGPJM6N7EOB7X27ZP2WM52HWSHT4Q, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.656204463 (542h19m51.527287509s) [noise=true , address=145.79.227.49:4370 , piece_id=QDMDBC3576X7ZYGUAC2DLJKWBNVOLSROCMA4UNFDZ47TTADILUAQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.656699927 (542h19m51.527782972s) [noise=true , address=45.91.116.9:4290 , piece_id=MP5DAA5UGHWQ3BA5R7GBWNIV2B4JYCL5WUSV2PY7GTO4PNLH2F2Q, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.657183808 (542h19m51.528266883s) [noise=true , address=77.90.31.18:4827 , piece_id=V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.657682177 (542h19m51.528765222s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.658706768 (542h19m51.529789813s) [noise=true , address=145.79.227.33:4876 , piece_id=E5QQDATADJON5NOOQ6X37GFKUQCFUKJRGK6G2EURBSUJVLF72Y7A, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.660113707 (542h19m51.531196752s) [noise=true , address=145.79.227.34:4915 , piece_id=MR5WOODFPXH57I6O2JCMC3EE5OWXF23CFLLATZXFP5XK6YPWDZJA, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.661356113 (542h19m51.532439158s) [noise=true , address=77.90.31.15:4727 , piece_id=3ACH3RUAR5KUXZU23AL3OQSGIZQOU5MFOFRN3JUYLQJBJCA4JOZQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.664431910 (542h19m51.535514955s) [noise=true , address=45.91.116.6:4998 , piece_id=FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.666377204 (542h19m51.537460249s) [noise=true , address=145.79.227.40:4067 , piece_id=36Q7DHHN5ELVD3BYSNUGJ7WYXNGIDKERTOPRAZJNJ7YQTOUPWEYQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.666754502 (542h19m51.537837548s) [noise=true , address=77.90.31.12:4594 , piece_id=DNXKJQOUHLAWMOCR35THP5CKCB6LZBW4GMJU6O6KKVSIJ3ALLONQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.668451284 (542h19m51.539534319s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.671076552 (542h19m51.542159587s) [noise=true , address=77.90.31.11:4560 , piece_id=HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.674568571 (542h19m51.545651626s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:52.998088476 (542h19m52.869171521s) [upload=475588]: Next returned nil segment
Dec 28 05:48:52.998103846 (542h19m52.869186881s) [upload=475588]: Waiting for error group managing segments...
Dec 28 05:48:53.543654582 (542h19m53.414737627s) [noise=true , address=145.79.227.40:4067 , piece_id=36Q7DHHN5ELVD3BYSNUGJ7WYXNGIDKERTOPRAZJNJ7YQTOUPWEYQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.543744133 (542h19m53.414827178s) [noise=true , address=145.79.227.49:4370 , piece_id=QDMDBC3576X7ZYGUAC2DLJKWBNVOLSROCMA4UNFDZ47TTADILUAQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.556254274 (542h19m53.427337309s) [noise=true , address=145.79.227.34:4915 , piece_id=MR5WOODFPXH57I6O2JCMC3EE5OWXF23CFLLATZXFP5XK6YPWDZJA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.810781113 (542h19m53.681864158s) [noise=true , address=77.90.31.13:4647 , piece_id=SOPU7ZUZQXRLU6TUL4L3ODX7U7WFAKUDOSK5UO66WMAIANU6PVUA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.814762255 (542h19m53.6858453s) [noise=true , address=77.90.31.16:4756 , piece_id=GY4LXIIIB7SVWA2WRAEHJVB6HLN6XQYISF7ZPVE4KA42C23MEQ4A, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.816966662 (542h19m53.688049707s) [noise=true , address=77.90.31.15:4727 , piece_id=3ACH3RUAR5KUXZU23AL3OQSGIZQOU5MFOFRN3JUYLQJBJCA4JOZQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.895077718 (542h19m53.766160783s) [noise=true , address=45.91.116.9:4290 , piece_id=MP5DAA5UGHWQ3BA5R7GBWNIV2B4JYCL5WUSV2PY7GTO4PNLH2F2Q, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.906586933 (542h19m53.777669978s) [noise=true , address=45.91.116.8:4059 , piece_id=OSA2YK4AH2DPQBNSJDFT3PLTGPJM6N7EOB7X27ZP2WM52HWSHT4Q, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.930785842 (542h19m53.801868877s) [noise=true , address=145.79.227.33:4876 , piece_id=E5QQDATADJON5NOOQ6X37GFKUQCFUKJRGK6G2EURBSUJVLF72Y7A, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.087446700 (542h19m53.958529735s) [noise=true , address=45.91.116.6:4998 , piece_id=FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.129981903 (542h19m54.001064938s) [noise=true , address=77.90.31.11:4560 , piece_id=HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.264740966 (542h19m54.135824011s) [noise=true , address=77.90.31.12:4594 , piece_id=DNXKJQOUHLAWMOCR35THP5CKCB6LZBW4GMJU6O6KKVSIJ3ALLONQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.368348401 (542h19m54.239431446s) [noise=true , address=77.90.31.18:4827 , piece_id=V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.437713885 (542h19m54.30879693s) [noise=true , address=145.79.227.44:4209 , piece_id=JRNDKXGHFY5OJYSY4ESURYMK5BNRDWY6INLHOWEK55KHMNRJO4AQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.437828132 (542h19m54.308911187s) [seg_pos={0 0}, upload=475588]: Segment reached optimal threshold of 14 pieces.
Dec 28 05:48:54.438064122 (542h19m54.309147167s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF
Dec 28 05:48:54.438074481 (542h19m54.309157546s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold
Dec 28 05:48:54.438099619 (542h19m54.309182664s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F4fJAGJM2ZPRt7vJH2cZmEbYeujmxcfi7aPZ2g9VPTNYBYRAi): protocol: expected piece hash; context canceled; EOF
Dec 28 05:48:54.438108586 (542h19m54.309191631s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold
Dec 28 05:48:54.438243403 (542h19m54.309326458s) [noise=true , address=77.90.31.19:4876 , piece_id=PUIQX7IZSNPKZTZTYH32KTFUM7K53KWZASLPT6JHAPMOB2JCXPGA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12H3UNBGeAufYSdVvEgYkv2QWeFtjKMvHvYGeqGeBvjEHqDDjBB): protocol: expected piece hash; context canceled; EOF
Dec 28 05:48:54.438252169 (542h19m54.309335214s) [noise=true , address=77.90.31.19:4876 , piece_id=PUIQX7IZSNPKZTZTYH32KTFUM7K53KWZASLPT6JHAPMOB2JCXPGA, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold
Dec 28 05:48:54.438391024 (542h19m54.309474069s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection
Dec 28 05:48:54.438398588 (542h19m54.309481633s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold
Dec 28 05:48:54.438699081 (542h19m54.309782126s) [seg_pos={0 0}, upload=475588]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 05:48:54.438794753 (542h19m54.309877798s) [upload=475588]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 05:48:54.438801255 (542h19m54.30988429s) [upload=475588]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 05:48:54.728435695 (542h19m54.59951876s) [upload=475588]: Done uploading segments...
Dec 28 05:48:54.728457938 (542h19m54.599540983s) [upload=475588]: Upload finished. err: <nil>
Duration: 48.722119698s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
26 out of 60 is failed (threshold=6) [XXX.....XXX....XXXX....XXX.X...X....X...XX.X..XXX....X..XX.X] duration:
5.001582917s trace:
https://jaeger.dev.storj.io/trace/1fac56088bbc7204
FAIL (2025-12-28 05:53:30 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001599489s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
25 out of 60 is failed (threshold=6) [.XXX.....XXX....XXXX....XXX.X...X....X...XX.X..XXX....X..XX.] duration:
4.095855284s trace:
https://jaeger.dev.storj.io/trace/6f497d94b2e58aeb
FAIL (2025-12-28 05:52:35 UTC)
orbiter warning: context deadline exceeded
Duration: 4.095885422s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [.....................X.........X...X....................X...] duration:
50.534694658s trace:
https://jaeger.dev.storj.io/trace/35a88c4b3a3d78b6
FAIL (2025-12-28 05:52:22 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH, Piece ID: HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA) context deadline exceeded; piecestore: (Node ID: 1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, Piece ID: GY4LXIIIB7SVWA2WRAEHJVB6HLN6XQYISF7ZPVE4KA42C23MEQ4A) context deadline exceeded; piecestore: (Node ID: 1RQ3vtArvKAvaMJTVa9VDDVdWR5oBuZ7gaPB3pzc1MouqBZhWu, Piece ID: FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ) context deadline exceeded; piecestore: (Node ID: 127KQ2X6VXddEeFRMau7MVGwtTsUmFsm8hTYDR2A6B4A1oVKfyo, Piece ID: V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ) context deadline exceeded
======LOGS======
) [noise=true , address=145.79.227.44:4209 , piece_id=JRNDKXGHFY5OJYSY4ESURYMK5BNRDWY6INLHOWEK55KHMNRJO4AQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.655023976 (542h19m51.526107021s) [noise=true , address=77.90.31.13:4647 , piece_id=SOPU7ZUZQXRLU6TUL4L3ODX7U7WFAKUDOSK5UO66WMAIANU6PVUA, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.655676448 (542h19m51.526759493s) [noise=true , address=45.91.116.8:4059 , piece_id=OSA2YK4AH2DPQBNSJDFT3PLTGPJM6N7EOB7X27ZP2WM52HWSHT4Q, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.656204463 (542h19m51.527287509s) [noise=true , address=145.79.227.49:4370 , piece_id=QDMDBC3576X7ZYGUAC2DLJKWBNVOLSROCMA4UNFDZ47TTADILUAQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.656699927 (542h19m51.527782972s) [noise=true , address=45.91.116.9:4290 , piece_id=MP5DAA5UGHWQ3BA5R7GBWNIV2B4JYCL5WUSV2PY7GTO4PNLH2F2Q, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.657183808 (542h19m51.528266883s) [noise=true , address=77.90.31.18:4827 , piece_id=V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.657682177 (542h19m51.528765222s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.658706768 (542h19m51.529789813s) [noise=true , address=145.79.227.33:4876 , piece_id=E5QQDATADJON5NOOQ6X37GFKUQCFUKJRGK6G2EURBSUJVLF72Y7A, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.660113707 (542h19m51.531196752s) [noise=true , address=145.79.227.34:4915 , piece_id=MR5WOODFPXH57I6O2JCMC3EE5OWXF23CFLLATZXFP5XK6YPWDZJA, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.661356113 (542h19m51.532439158s) [noise=true , address=77.90.31.15:4727 , piece_id=3ACH3RUAR5KUXZU23AL3OQSGIZQOU5MFOFRN3JUYLQJBJCA4JOZQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.664431910 (542h19m51.535514955s) [noise=true , address=45.91.116.6:4998 , piece_id=FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.666377204 (542h19m51.537460249s) [noise=true , address=145.79.227.40:4067 , piece_id=36Q7DHHN5ELVD3BYSNUGJ7WYXNGIDKERTOPRAZJNJ7YQTOUPWEYQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.666754502 (542h19m51.537837548s) [noise=true , address=77.90.31.12:4594 , piece_id=DNXKJQOUHLAWMOCR35THP5CKCB6LZBW4GMJU6O6KKVSIJ3ALLONQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.668451284 (542h19m51.539534319s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.671076552 (542h19m51.542159587s) [noise=true , address=77.90.31.11:4560 , piece_id=HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.674568571 (542h19m51.545651626s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:52.998088476 (542h19m52.869171521s) [upload=475588]: Next returned nil segment
Dec 28 05:48:52.998103846 (542h19m52.869186881s) [upload=475588]: Waiting for error group managing segments...
Dec 28 05:48:53.543654582 (542h19m53.414737627s) [noise=true , address=145.79.227.40:4067 , piece_id=36Q7DHHN5ELVD3BYSNUGJ7WYXNGIDKERTOPRAZJNJ7YQTOUPWEYQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.543744133 (542h19m53.414827178s) [noise=true , address=145.79.227.49:4370 , piece_id=QDMDBC3576X7ZYGUAC2DLJKWBNVOLSROCMA4UNFDZ47TTADILUAQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.556254274 (542h19m53.427337309s) [noise=true , address=145.79.227.34:4915 , piece_id=MR5WOODFPXH57I6O2JCMC3EE5OWXF23CFLLATZXFP5XK6YPWDZJA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.810781113 (542h19m53.681864158s) [noise=true , address=77.90.31.13:4647 , piece_id=SOPU7ZUZQXRLU6TUL4L3ODX7U7WFAKUDOSK5UO66WMAIANU6PVUA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.814762255 (542h19m53.6858453s) [noise=true , address=77.90.31.16:4756 , piece_id=GY4LXIIIB7SVWA2WRAEHJVB6HLN6XQYISF7ZPVE4KA42C23MEQ4A, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.816966662 (542h19m53.688049707s) [noise=true , address=77.90.31.15:4727 , piece_id=3ACH3RUAR5KUXZU23AL3OQSGIZQOU5MFOFRN3JUYLQJBJCA4JOZQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.895077718 (542h19m53.766160783s) [noise=true , address=45.91.116.9:4290 , piece_id=MP5DAA5UGHWQ3BA5R7GBWNIV2B4JYCL5WUSV2PY7GTO4PNLH2F2Q, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.906586933 (542h19m53.777669978s) [noise=true , address=45.91.116.8:4059 , piece_id=OSA2YK4AH2DPQBNSJDFT3PLTGPJM6N7EOB7X27ZP2WM52HWSHT4Q, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.930785842 (542h19m53.801868877s) [noise=true , address=145.79.227.33:4876 , piece_id=E5QQDATADJON5NOOQ6X37GFKUQCFUKJRGK6G2EURBSUJVLF72Y7A, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.087446700 (542h19m53.958529735s) [noise=true , address=45.91.116.6:4998 , piece_id=FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.129981903 (542h19m54.001064938s) [noise=true , address=77.90.31.11:4560 , piece_id=HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.264740966 (542h19m54.135824011s) [noise=true , address=77.90.31.12:4594 , piece_id=DNXKJQOUHLAWMOCR35THP5CKCB6LZBW4GMJU6O6KKVSIJ3ALLONQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.368348401 (542h19m54.239431446s) [noise=true , address=77.90.31.18:4827 , piece_id=V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.437713885 (542h19m54.30879693s) [noise=true , address=145.79.227.44:4209 , piece_id=JRNDKXGHFY5OJYSY4ESURYMK5BNRDWY6INLHOWEK55KHMNRJO4AQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.437828132 (542h19m54.308911187s) [seg_pos={0 0}, upload=475588]: Segment reached optimal threshold of 14 pieces.
Dec 28 05:48:54.438064122 (542h19m54.309147167s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF
Dec 28 05:48:54.438074481 (542h19m54.309157546s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold
Dec 28 05:48:54.438099619 (542h19m54.309182664s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F4fJAGJM2ZPRt7vJH2cZmEbYeujmxcfi7aPZ2g9VPTNYBYRAi): protocol: expected piece hash; context canceled; EOF
Dec 28 05:48:54.438108586 (542h19m54.309191631s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold
Dec 28 05:48:54.438243403 (542h19m54.309326458s) [noise=true , address=77.90.31.19:4876 , piece_id=PUIQX7IZSNPKZTZTYH32KTFUM7K53KWZASLPT6JHAPMOB2JCXPGA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12H3UNBGeAufYSdVvEgYkv2QWeFtjKMvHvYGeqGeBvjEHqDDjBB): protocol: expected piece hash; context canceled; EOF
Dec 28 05:48:54.438252169 (542h19m54.309335214s) [noise=true , address=77.90.31.19:4876 , piece_id=PUIQX7IZSNPKZTZTYH32KTFUM7K53KWZASLPT6JHAPMOB2JCXPGA, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold
Dec 28 05:48:54.438391024 (542h19m54.309474069s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection
Dec 28 05:48:54.438398588 (542h19m54.309481633s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold
Dec 28 05:48:54.438699081 (542h19m54.309782126s) [seg_pos={0 0}, upload=475588]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 05:48:54.438794753 (542h19m54.309877798s) [upload=475588]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 05:48:54.438801255 (542h19m54.30988429s) [upload=475588]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 05:48:54.728435695 (542h19m54.59951876s) [upload=475588]: Done uploading segments...
Dec 28 05:48:54.728457938 (542h19m54.599540983s) [upload=475588]: Upload finished. err: <nil>
Duration: 50.534708754s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
20 out of 60 is failed (threshold=3) [X...X.XX.X.XX..XX.X...X.X.XX....X...X.X..X.....X........X...] duration:
5.696687359s trace:
https://jaeger.dev.storj.io/trace/1c4c0fba90326f44
FAIL (2025-12-28 05:52:17 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: FBKTDMLHCINVRHMI64CCDXNLA7WWKW2MKGQKK2JNGK4TCJJR4VWA) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: IFFSYXZED5GVA2IKJLCXWZRPNZVYWAN63MGFDPTYSU7D4BMPT3RA) context deadline exceeded
======LOGS======
c 28 05:49:53.964236577 (542h20m53.835319622s) [upload=475605]: Starting upload
Dec 28 05:49:53.964304787 (542h20m53.835387822s) [upload=475605]: Done starting upload
Dec 28 05:49:53.964353770 (542h20m53.835436805s) [upload=475605]: Uploading segments...
Dec 28 05:49:53.964724576 (542h20m53.835807621s) [upload=475605]: Got next segment. Inline: false
Dec 28 05:49:53.964805681 (542h20m53.835888726s) [upload=475605]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 05:49:53.964813275 (542h20m53.83589632s) [upload=475605]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 05:49:54.210544977 (542h20m54.081628022s) [seg_pos={0 0}, upload=475605]: Begin upload segment...
Dec 28 05:49:54.210612576 (542h20m54.081695621s) [seg_pos={0 0}, upload=475605]: Done begin upload segment.
Dec 28 05:49:54.210655959 (542h20m54.081739004s) [noise=true , address=45.91.116.11:4118 , piece_id=FBKTDMLHCINVRHMI64CCDXNLA7WWKW2MKGQKK2JNGK4TCJJR4VWA, seg_pos={0 0}, upload=475605]: Uploading piece...
Dec 28 05:49:54.211437788 (542h20m54.082520833s) [noise=true , address=77.90.31.13:4626 , piece_id=RNJ7BCRCRYKKO2TCJIKEWAANA6PXFHGLYTWOM25X6TQ3LJCJPDMQ, seg_pos={0 0}, upload=475605]: Uploading piece...
Dec 28 05:49:54.212066334 (542h20m54.08314938s) [noise=true , address=77.90.31.18:4805 , piece_id=IFFSYXZED5GVA2IKJLCXWZRPNZVYWAN63MGFDPTYSU7D4BMPT3RA, seg_pos={0 0}, upload=475605]: Uploading piece...
Dec 28 05:49:54.212482277 (542h20m54.083565322s) [noise=true , address=145.79.227.36:4518 , piece_id=NA3L2ESKHDDCJBJHGCJZADLDMTZ5ZZXE2FEZ6GJ4BWEVPRX6U4OQ, seg_pos={0 0}, upload=475605]: Uploading piece...
Dec 28 05:49:54.213019269 (542h20m54.084102314s) [noise=true , address=145.79.227.50:4374 , piece_id=4RKQXOQYWRM7ZC3JQFOILXWWDHPYFZYR3YBJLZRA5RZB2PNVMLKQ, seg_pos={0 0}, upload=475605]: Uploading piece...
Dec 28 05:49:54.213518980 (542h20m54.084602015s) [noise=true , address=145.79.227.51:4411 , piece_id=7Z7WIROUYCJMMISCUSHSN7WCRDSCEWOSLADDDLGK7FZIXHQZE4AA, seg_pos={0 0}, upload=475605]: Uploading piece...
Dec 28 05:49:54.214019624 (542h20m54.085102659s) [noise=true , address=145.79.227.7:4293 , piece_id=VH6QJUZI2P2S3E5CFD3CMY5TOOUFX7P2DOX2LCZT7BHC7GVGKD6Q, seg_pos={0 0}, upload=475605]: Uploading piece...
Dec 28 05:49:54.214565593 (542h20m54.085648638s) [noise=true , address=145.79.227.33:4870 , piece_id=4U6WHLGVALEVHAJOFXG6D7HZ4IU6G5BYOFTGTPEMHQ2G3DK4O2SA, seg_pos={0 0}, upload=475605]: Uploading piece...
Dec 28 05:49:54.216167082 (542h20m54.087250127s) [noise=true , address=45.91.116.13:4155 , piece_id=P3BMQU5EHE7BLVM5XRKDPB2HQIDELVE67QXQ4TWCYSB5DQRVOYOA, seg_pos={0 0}, upload=475605]: Uploading piece...
Dec 28 05:49:54.217939377 (542h20m54.089022432s) [noise=true , address=77.90.31.14:4662 , piece_id=JOCXDUJA3YYWIYVZEOMDAXSX7A7XFAARLYZK5GTE5UKHUN6SKZVA, seg_pos={0 0}, upload=475605]: Uploading piece...
Dec 28 05:49:54.219536669 (542h20m54.090619714s) [noise=true , address=145.79.227.2:4113 , piece_id=H37EOIGKBSNPVYRPD7LZV6AOQPRBREXGDCRZVFZRCFTC4FKAEAQA, seg_pos={0 0}, upload=475605]: Uploading piece...
Dec 28 05:49:54.221295528 (542h20m54.092378573s) [noise=true , address=145.79.227.6:4258 , piece_id=2T2PHFQUXIUYLV54KLS4E5V3FBS7UFGGKJ2AJX7OHVWMGG5US3EQ, seg_pos={0 0}, upload=475605]: Uploading piece...
Dec 28 05:49:54.223541675 (542h20m54.09462471s) [noise=true , address=145.79.227.52:4446 , piece_id=RDJSEQUNF4ESGJMW3OEKEQQF3CCUGRFLOB5TS6T6HF6X4ZFOTOUA, seg_pos={0 0}, upload=475605]: Uploading piece...
Dec 28 05:49:54.225988124 (542h20m54.097071169s) [noise=true , address=145.79.227.48:4303 , piece_id=2C5QMP3KKEGDAEVALCCSJ5KFD4RFY4CNGRVXJMJGNXUI2YYUNXXQ, seg_pos={0 0}, upload=475605]: Uploading piece...
Dec 28 05:49:54.228586531 (542h20m54.099669566s) [noise=true , address=145.79.227.37:4565 , piece_id=ZHTVIULJSWVWH4TKMOGBL3ATODW7OS7JZMYDO5QKZK3Q2AVEWTLQ, seg_pos={0 0}, upload=475605]: Uploading piece...
Dec 28 05:49:54.231333522 (542h20m54.102416567s) [noise=true , address=145.79.227.42:4123 , piece_id=DRQME77DJN6KVBAGGSOZMLGPKGYYTHSFXMXGPSL43IXIGM5AJWBA, seg_pos={0 0}, upload=475605]: Uploading piece...
Dec 28 05:49:55.849579995 (542h20m55.72066303s) [upload=475605]: Next returned nil segment
Dec 28 05:49:55.849594874 (542h20m55.720677909s) [upload=475605]: Waiting for error group managing segments...
Dec 28 05:49:56.246089673 (542h20m56.117172718s) [noise=true , address=145.79.227.42:4123 , piece_id=DRQME77DJN6KVBAGGSOZMLGPKGYYTHSFXMXGPSL43IXIGM5AJWBA, seg_pos={0 0}, upload=475605]: Done uploading piece. err: <nil>
Dec 28 05:49:56.247565743 (542h20m56.118648789s) [noise=true , address=145.79.227.6:4258 , piece_id=2T2PHFQUXIUYLV54KLS4E5V3FBS7UFGGKJ2AJX7OHVWMGG5US3EQ, seg_pos={0 0}, upload=475605]: Done uploading piece. err: <nil>
Dec 28 05:49:56.249641577 (542h20m56.120724622s) [noise=true , address=145.79.227.2:4113 , piece_id=H37EOIGKBSNPVYRPD7LZV6AOQPRBREXGDCRZVFZRCFTC4FKAEAQA, seg_pos={0 0}, upload=475605]: Done uploading piece. err: <nil>
Dec 28 05:49:56.251345281 (542h20m56.122428326s) [noise=true , address=145.79.227.36:4518 , piece_id=NA3L2ESKHDDCJBJHGCJZADLDMTZ5ZZXE2FEZ6GJ4BWEVPRX6U4OQ, seg_pos={0 0}, upload=475605]: Done uploading piece. err: <nil>
Dec 28 05:49:56.254033299 (542h20m56.125116354s) [noise=true , address=145.79.227.50:4374 , piece_id=4RKQXOQYWRM7ZC3JQFOILXWWDHPYFZYR3YBJLZRA5RZB2PNVMLKQ, seg_pos={0 0}, upload=475605]: Done uploading piece. err: <nil>
Dec 28 05:49:56.258547275 (542h20m56.12963032s) [noise=true , address=145.79.227.52:4446 , piece_id=RDJSEQUNF4ESGJMW3OEKEQQF3CCUGRFLOB5TS6T6HF6X4ZFOTOUA, seg_pos={0 0}, upload=475605]: Done uploading piece. err: <nil>
Dec 28 05:49:56.261965754 (542h20m56.133048789s) [noise=true , address=145.79.227.51:4411 , piece_id=7Z7WIROUYCJMMISCUSHSN7WCRDSCEWOSLADDDLGK7FZIXHQZE4AA, seg_pos={0 0}, upload=475605]: Done uploading piece. err: <nil>
Dec 28 05:49:56.262566037 (542h20m56.133649082s) [noise=true , address=145.79.227.48:4303 , piece_id=2C5QMP3KKEGDAEVALCCSJ5KFD4RFY4CNGRVXJMJGNXUI2YYUNXXQ, seg_pos={0 0}, upload=475605]: Done uploading piece. err: <nil>
Dec 28 05:49:56.262842934 (542h20m56.133925979s) [noise=true , address=145.79.227.37:4565 , piece_id=ZHTVIULJSWVWH4TKMOGBL3ATODW7OS7JZMYDO5QKZK3Q2AVEWTLQ, seg_pos={0 0}, upload=475605]: Done uploading piece. err: <nil>
Dec 28 05:49:56.263743830 (542h20m56.134826875s) [noise=true , address=145.79.227.7:4293 , piece_id=VH6QJUZI2P2S3E5CFD3CMY5TOOUFX7P2DOX2LCZT7BHC7GVGKD6Q, seg_pos={0 0}, upload=475605]: Done uploading piece. err: <nil>
Dec 28 05:49:56.521742791 (542h20m56.392825866s) [noise=true , address=45.91.116.11:4118 , piece_id=FBKTDMLHCINVRHMI64CCDXNLA7WWKW2MKGQKK2JNGK4TCJJR4VWA, seg_pos={0 0}, upload=475605]: Done uploading piece. err: <nil>
Dec 28 05:49:56.532519422 (542h20m56.403602457s) [noise=true , address=77.90.31.18:4805 , piece_id=IFFSYXZED5GVA2IKJLCXWZRPNZVYWAN63MGFDPTYSU7D4BMPT3RA, seg_pos={0 0}, upload=475605]: Done uploading piece. err: <nil>
Dec 28 05:49:56.621882173 (542h20m56.492965218s) [noise=true , address=45.91.116.13:4155 , piece_id=P3BMQU5EHE7BLVM5XRKDPB2HQIDELVE67QXQ4TWCYSB5DQRVOYOA, seg_pos={0 0}, upload=475605]: Done uploading piece. err: <nil>
Dec 28 05:49:56.621906769 (542h20m56.492989814s) [seg_pos={0 0}, upload=475605]: Segment reached optimal threshold of 13 pieces.
Dec 28 05:49:56.622133100 (542h20m56.493216135s) [noise=true , address=145.79.227.33:4870 , piece_id=4U6WHLGVALEVHAJOFXG6D7HZ4IU6G5BYOFTGTPEMHQ2G3DK4O2SA, seg_pos={0 0}, upload=475605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF
Dec 28 05:49:56.622143089 (542h20m56.493226124s) [noise=true , address=145.79.227.33:4870 , piece_id=4U6WHLGVALEVHAJOFXG6D7HZ4IU6G5BYOFTGTPEMHQ2G3DK4O2SA, seg_pos={0 0}, upload=475605]: Long tail cancellation due to optimal threshold
Dec 28 05:49:56.622361354 (542h20m56.493444399s) [noise=true , address=77.90.31.14:4662 , piece_id=JOCXDUJA3YYWIYVZEOMDAXSX7A7XFAARLYZK5GTE5UKHUN6SKZVA, seg_pos={0 0}, upload=475605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:55736->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:55736->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55736->77.90.31.14:4662: use of closed network connection
Dec 28 05:49:56.622367477 (542h20m56.493450512s) [noise=true , address=77.90.31.14:4662 , piece_id=JOCXDUJA3YYWIYVZEOMDAXSX7A7XFAARLYZK5GTE5UKHUN6SKZVA, seg_pos={0 0}, upload=475605]: Long tail cancellation due to optimal threshold
Dec 28 05:49:56.622387384 (542h20m56.493470429s) [noise=true , address=77.90.31.13:4626 , piece_id=RNJ7BCRCRYKKO2TCJIKEWAANA6PXFHGLYTWOM25X6TQ3LJCJPDMQ, seg_pos={0 0}, upload=475605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF
Dec 28 05:49:56.622408304 (542h20m56.49349135s) [noise=true , address=77.90.31.13:4626 , piece_id=RNJ7BCRCRYKKO2TCJIKEWAANA6PXFHGLYTWOM25X6TQ3LJCJPDMQ, seg_pos={0 0}, upload=475605]: Long tail cancellation due to optimal threshold
Dec 28 05:49:56.622549654 (542h20m56.493632689s) [seg_pos={0 0}, upload=475605]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 05:49:56.622626440 (542h20m56.493709475s) [upload=475605]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 05:49:56.622631609 (542h20m56.493714654s) [upload=475605]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 05:49:56.919679987 (542h20m56.790763083s) [upload=475605]: Done uploading segments...
Dec 28 05:49:56.919701469 (542h20m56.790784504s) [upload=475605]: Upload finished. err: <nil>
Duration: 5.69671971s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
26 out of 60 is failed (threshold=6) [X.XXX.....XXX....XXXX....XXX.X...X....X...XX.X..XXX....X..XX] duration:
5.001256015s trace:
https://jaeger.dev.storj.io/trace/6f497d94b2e58aeb
FAIL (2025-12-28 05:51:55 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001281423s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [......................X.........X...X....................X..] duration:
45.486778383s trace:
https://jaeger.dev.storj.io/trace/35a88c4b3a3d78b6
FAIL (2025-12-28 05:51:31 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH, Piece ID: HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA) context deadline exceeded; piecestore: (Node ID: 1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, Piece ID: GY4LXIIIB7SVWA2WRAEHJVB6HLN6XQYISF7ZPVE4KA42C23MEQ4A) context deadline exceeded; piecestore: (Node ID: 1RQ3vtArvKAvaMJTVa9VDDVdWR5oBuZ7gaPB3pzc1MouqBZhWu, Piece ID: FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ) context deadline exceeded; piecestore: (Node ID: 127KQ2X6VXddEeFRMau7MVGwtTsUmFsm8hTYDR2A6B4A1oVKfyo, Piece ID: V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ) context deadline exceeded
======LOGS======
) [noise=true , address=145.79.227.44:4209 , piece_id=JRNDKXGHFY5OJYSY4ESURYMK5BNRDWY6INLHOWEK55KHMNRJO4AQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.655023976 (542h19m51.526107021s) [noise=true , address=77.90.31.13:4647 , piece_id=SOPU7ZUZQXRLU6TUL4L3ODX7U7WFAKUDOSK5UO66WMAIANU6PVUA, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.655676448 (542h19m51.526759493s) [noise=true , address=45.91.116.8:4059 , piece_id=OSA2YK4AH2DPQBNSJDFT3PLTGPJM6N7EOB7X27ZP2WM52HWSHT4Q, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.656204463 (542h19m51.527287509s) [noise=true , address=145.79.227.49:4370 , piece_id=QDMDBC3576X7ZYGUAC2DLJKWBNVOLSROCMA4UNFDZ47TTADILUAQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.656699927 (542h19m51.527782972s) [noise=true , address=45.91.116.9:4290 , piece_id=MP5DAA5UGHWQ3BA5R7GBWNIV2B4JYCL5WUSV2PY7GTO4PNLH2F2Q, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.657183808 (542h19m51.528266883s) [noise=true , address=77.90.31.18:4827 , piece_id=V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.657682177 (542h19m51.528765222s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.658706768 (542h19m51.529789813s) [noise=true , address=145.79.227.33:4876 , piece_id=E5QQDATADJON5NOOQ6X37GFKUQCFUKJRGK6G2EURBSUJVLF72Y7A, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.660113707 (542h19m51.531196752s) [noise=true , address=145.79.227.34:4915 , piece_id=MR5WOODFPXH57I6O2JCMC3EE5OWXF23CFLLATZXFP5XK6YPWDZJA, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.661356113 (542h19m51.532439158s) [noise=true , address=77.90.31.15:4727 , piece_id=3ACH3RUAR5KUXZU23AL3OQSGIZQOU5MFOFRN3JUYLQJBJCA4JOZQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.664431910 (542h19m51.535514955s) [noise=true , address=45.91.116.6:4998 , piece_id=FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.666377204 (542h19m51.537460249s) [noise=true , address=145.79.227.40:4067 , piece_id=36Q7DHHN5ELVD3BYSNUGJ7WYXNGIDKERTOPRAZJNJ7YQTOUPWEYQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.666754502 (542h19m51.537837548s) [noise=true , address=77.90.31.12:4594 , piece_id=DNXKJQOUHLAWMOCR35THP5CKCB6LZBW4GMJU6O6KKVSIJ3ALLONQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.668451284 (542h19m51.539534319s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.671076552 (542h19m51.542159587s) [noise=true , address=77.90.31.11:4560 , piece_id=HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.674568571 (542h19m51.545651626s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:52.998088476 (542h19m52.869171521s) [upload=475588]: Next returned nil segment
Dec 28 05:48:52.998103846 (542h19m52.869186881s) [upload=475588]: Waiting for error group managing segments...
Dec 28 05:48:53.543654582 (542h19m53.414737627s) [noise=true , address=145.79.227.40:4067 , piece_id=36Q7DHHN5ELVD3BYSNUGJ7WYXNGIDKERTOPRAZJNJ7YQTOUPWEYQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.543744133 (542h19m53.414827178s) [noise=true , address=145.79.227.49:4370 , piece_id=QDMDBC3576X7ZYGUAC2DLJKWBNVOLSROCMA4UNFDZ47TTADILUAQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.556254274 (542h19m53.427337309s) [noise=true , address=145.79.227.34:4915 , piece_id=MR5WOODFPXH57I6O2JCMC3EE5OWXF23CFLLATZXFP5XK6YPWDZJA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.810781113 (542h19m53.681864158s) [noise=true , address=77.90.31.13:4647 , piece_id=SOPU7ZUZQXRLU6TUL4L3ODX7U7WFAKUDOSK5UO66WMAIANU6PVUA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.814762255 (542h19m53.6858453s) [noise=true , address=77.90.31.16:4756 , piece_id=GY4LXIIIB7SVWA2WRAEHJVB6HLN6XQYISF7ZPVE4KA42C23MEQ4A, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.816966662 (542h19m53.688049707s) [noise=true , address=77.90.31.15:4727 , piece_id=3ACH3RUAR5KUXZU23AL3OQSGIZQOU5MFOFRN3JUYLQJBJCA4JOZQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.895077718 (542h19m53.766160783s) [noise=true , address=45.91.116.9:4290 , piece_id=MP5DAA5UGHWQ3BA5R7GBWNIV2B4JYCL5WUSV2PY7GTO4PNLH2F2Q, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.906586933 (542h19m53.777669978s) [noise=true , address=45.91.116.8:4059 , piece_id=OSA2YK4AH2DPQBNSJDFT3PLTGPJM6N7EOB7X27ZP2WM52HWSHT4Q, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.930785842 (542h19m53.801868877s) [noise=true , address=145.79.227.33:4876 , piece_id=E5QQDATADJON5NOOQ6X37GFKUQCFUKJRGK6G2EURBSUJVLF72Y7A, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.087446700 (542h19m53.958529735s) [noise=true , address=45.91.116.6:4998 , piece_id=FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.129981903 (542h19m54.001064938s) [noise=true , address=77.90.31.11:4560 , piece_id=HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.264740966 (542h19m54.135824011s) [noise=true , address=77.90.31.12:4594 , piece_id=DNXKJQOUHLAWMOCR35THP5CKCB6LZBW4GMJU6O6KKVSIJ3ALLONQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.368348401 (542h19m54.239431446s) [noise=true , address=77.90.31.18:4827 , piece_id=V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.437713885 (542h19m54.30879693s) [noise=true , address=145.79.227.44:4209 , piece_id=JRNDKXGHFY5OJYSY4ESURYMK5BNRDWY6INLHOWEK55KHMNRJO4AQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.437828132 (542h19m54.308911187s) [seg_pos={0 0}, upload=475588]: Segment reached optimal threshold of 14 pieces.
Dec 28 05:48:54.438064122 (542h19m54.309147167s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF
Dec 28 05:48:54.438074481 (542h19m54.309157546s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold
Dec 28 05:48:54.438099619 (542h19m54.309182664s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F4fJAGJM2ZPRt7vJH2cZmEbYeujmxcfi7aPZ2g9VPTNYBYRAi): protocol: expected piece hash; context canceled; EOF
Dec 28 05:48:54.438108586 (542h19m54.309191631s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold
Dec 28 05:48:54.438243403 (542h19m54.309326458s) [noise=true , address=77.90.31.19:4876 , piece_id=PUIQX7IZSNPKZTZTYH32KTFUM7K53KWZASLPT6JHAPMOB2JCXPGA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12H3UNBGeAufYSdVvEgYkv2QWeFtjKMvHvYGeqGeBvjEHqDDjBB): protocol: expected piece hash; context canceled; EOF
Dec 28 05:48:54.438252169 (542h19m54.309335214s) [noise=true , address=77.90.31.19:4876 , piece_id=PUIQX7IZSNPKZTZTYH32KTFUM7K53KWZASLPT6JHAPMOB2JCXPGA, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold
Dec 28 05:48:54.438391024 (542h19m54.309474069s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection
Dec 28 05:48:54.438398588 (542h19m54.309481633s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold
Dec 28 05:48:54.438699081 (542h19m54.309782126s) [seg_pos={0 0}, upload=475588]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 05:48:54.438794753 (542h19m54.309877798s) [upload=475588]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 05:48:54.438801255 (542h19m54.30988429s) [upload=475588]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 05:48:54.728435695 (542h19m54.59951876s) [upload=475588]: Done uploading segments...
Dec 28 05:48:54.728457938 (542h19m54.599540983s) [upload=475588]: Upload finished. err: <nil>
Duration: 45.486803589s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
20 out of 60 is failed (threshold=3) [.X...X.XX.X.XX..XX.X...X.X.XX....X...X.X..X.....X........X..] duration:
5.94234465s trace:
https://jaeger.dev.storj.io/trace/1c4c0fba90326f44
FAIL (2025-12-28 05:51:27 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: FBKTDMLHCINVRHMI64CCDXNLA7WWKW2MKGQKK2JNGK4TCJJR4VWA) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: IFFSYXZED5GVA2IKJLCXWZRPNZVYWAN63MGFDPTYSU7D4BMPT3RA) context deadline exceeded
======LOGS======
c 28 05:49:53.964236577 (542h20m53.835319622s) [upload=475605]: Starting upload
Dec 28 05:49:53.964304787 (542h20m53.835387822s) [upload=475605]: Done starting upload
Dec 28 05:49:53.964353770 (542h20m53.835436805s) [upload=475605]: Uploading segments...
Dec 28 05:49:53.964724576 (542h20m53.835807621s) [upload=475605]: Got next segment. Inline: false
Dec 28 05:49:53.964805681 (542h20m53.835888726s) [upload=475605]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 05:49:53.964813275 (542h20m53.83589632s) [upload=475605]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 05:49:54.210544977 (542h20m54.081628022s) [seg_pos={0 0}, upload=475605]: Begin upload segment...
Dec 28 05:49:54.210612576 (542h20m54.081695621s) [seg_pos={0 0}, upload=475605]: Done begin upload segment.
Dec 28 05:49:54.210655959 (542h20m54.081739004s) [noise=true , address=45.91.116.11:4118 , piece_id=FBKTDMLHCINVRHMI64CCDXNLA7WWKW2MKGQKK2JNGK4TCJJR4VWA, seg_pos={0 0}, upload=475605]: Uploading piece...
Dec 28 05:49:54.211437788 (542h20m54.082520833s) [noise=true , address=77.90.31.13:4626 , piece_id=RNJ7BCRCRYKKO2TCJIKEWAANA6PXFHGLYTWOM25X6TQ3LJCJPDMQ, seg_pos={0 0}, upload=475605]: Uploading piece...
Dec 28 05:49:54.212066334 (542h20m54.08314938s) [noise=true , address=77.90.31.18:4805 , piece_id=IFFSYXZED5GVA2IKJLCXWZRPNZVYWAN63MGFDPTYSU7D4BMPT3RA, seg_pos={0 0}, upload=475605]: Uploading piece...
Dec 28 05:49:54.212482277 (542h20m54.083565322s) [noise=true , address=145.79.227.36:4518 , piece_id=NA3L2ESKHDDCJBJHGCJZADLDMTZ5ZZXE2FEZ6GJ4BWEVPRX6U4OQ, seg_pos={0 0}, upload=475605]: Uploading piece...
Dec 28 05:49:54.213019269 (542h20m54.084102314s) [noise=true , address=145.79.227.50:4374 , piece_id=4RKQXOQYWRM7ZC3JQFOILXWWDHPYFZYR3YBJLZRA5RZB2PNVMLKQ, seg_pos={0 0}, upload=475605]: Uploading piece...
Dec 28 05:49:54.213518980 (542h20m54.084602015s) [noise=true , address=145.79.227.51:4411 , piece_id=7Z7WIROUYCJMMISCUSHSN7WCRDSCEWOSLADDDLGK7FZIXHQZE4AA, seg_pos={0 0}, upload=475605]: Uploading piece...
Dec 28 05:49:54.214019624 (542h20m54.085102659s) [noise=true , address=145.79.227.7:4293 , piece_id=VH6QJUZI2P2S3E5CFD3CMY5TOOUFX7P2DOX2LCZT7BHC7GVGKD6Q, seg_pos={0 0}, upload=475605]: Uploading piece...
Dec 28 05:49:54.214565593 (542h20m54.085648638s) [noise=true , address=145.79.227.33:4870 , piece_id=4U6WHLGVALEVHAJOFXG6D7HZ4IU6G5BYOFTGTPEMHQ2G3DK4O2SA, seg_pos={0 0}, upload=475605]: Uploading piece...
Dec 28 05:49:54.216167082 (542h20m54.087250127s) [noise=true , address=45.91.116.13:4155 , piece_id=P3BMQU5EHE7BLVM5XRKDPB2HQIDELVE67QXQ4TWCYSB5DQRVOYOA, seg_pos={0 0}, upload=475605]: Uploading piece...
Dec 28 05:49:54.217939377 (542h20m54.089022432s) [noise=true , address=77.90.31.14:4662 , piece_id=JOCXDUJA3YYWIYVZEOMDAXSX7A7XFAARLYZK5GTE5UKHUN6SKZVA, seg_pos={0 0}, upload=475605]: Uploading piece...
Dec 28 05:49:54.219536669 (542h20m54.090619714s) [noise=true , address=145.79.227.2:4113 , piece_id=H37EOIGKBSNPVYRPD7LZV6AOQPRBREXGDCRZVFZRCFTC4FKAEAQA, seg_pos={0 0}, upload=475605]: Uploading piece...
Dec 28 05:49:54.221295528 (542h20m54.092378573s) [noise=true , address=145.79.227.6:4258 , piece_id=2T2PHFQUXIUYLV54KLS4E5V3FBS7UFGGKJ2AJX7OHVWMGG5US3EQ, seg_pos={0 0}, upload=475605]: Uploading piece...
Dec 28 05:49:54.223541675 (542h20m54.09462471s) [noise=true , address=145.79.227.52:4446 , piece_id=RDJSEQUNF4ESGJMW3OEKEQQF3CCUGRFLOB5TS6T6HF6X4ZFOTOUA, seg_pos={0 0}, upload=475605]: Uploading piece...
Dec 28 05:49:54.225988124 (542h20m54.097071169s) [noise=true , address=145.79.227.48:4303 , piece_id=2C5QMP3KKEGDAEVALCCSJ5KFD4RFY4CNGRVXJMJGNXUI2YYUNXXQ, seg_pos={0 0}, upload=475605]: Uploading piece...
Dec 28 05:49:54.228586531 (542h20m54.099669566s) [noise=true , address=145.79.227.37:4565 , piece_id=ZHTVIULJSWVWH4TKMOGBL3ATODW7OS7JZMYDO5QKZK3Q2AVEWTLQ, seg_pos={0 0}, upload=475605]: Uploading piece...
Dec 28 05:49:54.231333522 (542h20m54.102416567s) [noise=true , address=145.79.227.42:4123 , piece_id=DRQME77DJN6KVBAGGSOZMLGPKGYYTHSFXMXGPSL43IXIGM5AJWBA, seg_pos={0 0}, upload=475605]: Uploading piece...
Dec 28 05:49:55.849579995 (542h20m55.72066303s) [upload=475605]: Next returned nil segment
Dec 28 05:49:55.849594874 (542h20m55.720677909s) [upload=475605]: Waiting for error group managing segments...
Dec 28 05:49:56.246089673 (542h20m56.117172718s) [noise=true , address=145.79.227.42:4123 , piece_id=DRQME77DJN6KVBAGGSOZMLGPKGYYTHSFXMXGPSL43IXIGM5AJWBA, seg_pos={0 0}, upload=475605]: Done uploading piece. err: <nil>
Dec 28 05:49:56.247565743 (542h20m56.118648789s) [noise=true , address=145.79.227.6:4258 , piece_id=2T2PHFQUXIUYLV54KLS4E5V3FBS7UFGGKJ2AJX7OHVWMGG5US3EQ, seg_pos={0 0}, upload=475605]: Done uploading piece. err: <nil>
Dec 28 05:49:56.249641577 (542h20m56.120724622s) [noise=true , address=145.79.227.2:4113 , piece_id=H37EOIGKBSNPVYRPD7LZV6AOQPRBREXGDCRZVFZRCFTC4FKAEAQA, seg_pos={0 0}, upload=475605]: Done uploading piece. err: <nil>
Dec 28 05:49:56.251345281 (542h20m56.122428326s) [noise=true , address=145.79.227.36:4518 , piece_id=NA3L2ESKHDDCJBJHGCJZADLDMTZ5ZZXE2FEZ6GJ4BWEVPRX6U4OQ, seg_pos={0 0}, upload=475605]: Done uploading piece. err: <nil>
Dec 28 05:49:56.254033299 (542h20m56.125116354s) [noise=true , address=145.79.227.50:4374 , piece_id=4RKQXOQYWRM7ZC3JQFOILXWWDHPYFZYR3YBJLZRA5RZB2PNVMLKQ, seg_pos={0 0}, upload=475605]: Done uploading piece. err: <nil>
Dec 28 05:49:56.258547275 (542h20m56.12963032s) [noise=true , address=145.79.227.52:4446 , piece_id=RDJSEQUNF4ESGJMW3OEKEQQF3CCUGRFLOB5TS6T6HF6X4ZFOTOUA, seg_pos={0 0}, upload=475605]: Done uploading piece. err: <nil>
Dec 28 05:49:56.261965754 (542h20m56.133048789s) [noise=true , address=145.79.227.51:4411 , piece_id=7Z7WIROUYCJMMISCUSHSN7WCRDSCEWOSLADDDLGK7FZIXHQZE4AA, seg_pos={0 0}, upload=475605]: Done uploading piece. err: <nil>
Dec 28 05:49:56.262566037 (542h20m56.133649082s) [noise=true , address=145.79.227.48:4303 , piece_id=2C5QMP3KKEGDAEVALCCSJ5KFD4RFY4CNGRVXJMJGNXUI2YYUNXXQ, seg_pos={0 0}, upload=475605]: Done uploading piece. err: <nil>
Dec 28 05:49:56.262842934 (542h20m56.133925979s) [noise=true , address=145.79.227.37:4565 , piece_id=ZHTVIULJSWVWH4TKMOGBL3ATODW7OS7JZMYDO5QKZK3Q2AVEWTLQ, seg_pos={0 0}, upload=475605]: Done uploading piece. err: <nil>
Dec 28 05:49:56.263743830 (542h20m56.134826875s) [noise=true , address=145.79.227.7:4293 , piece_id=VH6QJUZI2P2S3E5CFD3CMY5TOOUFX7P2DOX2LCZT7BHC7GVGKD6Q, seg_pos={0 0}, upload=475605]: Done uploading piece. err: <nil>
Dec 28 05:49:56.521742791 (542h20m56.392825866s) [noise=true , address=45.91.116.11:4118 , piece_id=FBKTDMLHCINVRHMI64CCDXNLA7WWKW2MKGQKK2JNGK4TCJJR4VWA, seg_pos={0 0}, upload=475605]: Done uploading piece. err: <nil>
Dec 28 05:49:56.532519422 (542h20m56.403602457s) [noise=true , address=77.90.31.18:4805 , piece_id=IFFSYXZED5GVA2IKJLCXWZRPNZVYWAN63MGFDPTYSU7D4BMPT3RA, seg_pos={0 0}, upload=475605]: Done uploading piece. err: <nil>
Dec 28 05:49:56.621882173 (542h20m56.492965218s) [noise=true , address=45.91.116.13:4155 , piece_id=P3BMQU5EHE7BLVM5XRKDPB2HQIDELVE67QXQ4TWCYSB5DQRVOYOA, seg_pos={0 0}, upload=475605]: Done uploading piece. err: <nil>
Dec 28 05:49:56.621906769 (542h20m56.492989814s) [seg_pos={0 0}, upload=475605]: Segment reached optimal threshold of 13 pieces.
Dec 28 05:49:56.622133100 (542h20m56.493216135s) [noise=true , address=145.79.227.33:4870 , piece_id=4U6WHLGVALEVHAJOFXG6D7HZ4IU6G5BYOFTGTPEMHQ2G3DK4O2SA, seg_pos={0 0}, upload=475605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF
Dec 28 05:49:56.622143089 (542h20m56.493226124s) [noise=true , address=145.79.227.33:4870 , piece_id=4U6WHLGVALEVHAJOFXG6D7HZ4IU6G5BYOFTGTPEMHQ2G3DK4O2SA, seg_pos={0 0}, upload=475605]: Long tail cancellation due to optimal threshold
Dec 28 05:49:56.622361354 (542h20m56.493444399s) [noise=true , address=77.90.31.14:4662 , piece_id=JOCXDUJA3YYWIYVZEOMDAXSX7A7XFAARLYZK5GTE5UKHUN6SKZVA, seg_pos={0 0}, upload=475605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:55736->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:55736->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55736->77.90.31.14:4662: use of closed network connection
Dec 28 05:49:56.622367477 (542h20m56.493450512s) [noise=true , address=77.90.31.14:4662 , piece_id=JOCXDUJA3YYWIYVZEOMDAXSX7A7XFAARLYZK5GTE5UKHUN6SKZVA, seg_pos={0 0}, upload=475605]: Long tail cancellation due to optimal threshold
Dec 28 05:49:56.622387384 (542h20m56.493470429s) [noise=true , address=77.90.31.13:4626 , piece_id=RNJ7BCRCRYKKO2TCJIKEWAANA6PXFHGLYTWOM25X6TQ3LJCJPDMQ, seg_pos={0 0}, upload=475605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF
Dec 28 05:49:56.622408304 (542h20m56.49349135s) [noise=true , address=77.90.31.13:4626 , piece_id=RNJ7BCRCRYKKO2TCJIKEWAANA6PXFHGLYTWOM25X6TQ3LJCJPDMQ, seg_pos={0 0}, upload=475605]: Long tail cancellation due to optimal threshold
Dec 28 05:49:56.622549654 (542h20m56.493632689s) [seg_pos={0 0}, upload=475605]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 05:49:56.622626440 (542h20m56.493709475s) [upload=475605]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 05:49:56.622631609 (542h20m56.493714654s) [upload=475605]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 05:49:56.919679987 (542h20m56.790763083s) [upload=475605]: Done uploading segments...
Dec 28 05:49:56.919701469 (542h20m56.790784504s) [upload=475605]: Upload finished. err: <nil>
Duration: 5.942373133s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
26 out of 60 is failed (threshold=6) [XX.XXX.....XXX....XXXX....XXX.X...X....X...XX.X..XXX....X..X] duration:
5.001570023s trace:
https://jaeger.dev.storj.io/trace/58036b7cef0dbc6a
FAIL (2025-12-28 05:50:59 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001595782s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
25 out of 60 is failed (threshold=6) [.XX.XXX.....XXX....XXXX....XXX.X...X....X...XX.X..XXX....X..] duration:
4.633930586s trace:
https://jaeger.dev.storj.io/trace/4fb94e1d2f36d9ac
FAIL (2025-12-28 05:50:39 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.633969882s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
20 out of 60 is failed (threshold=3) [..X...X.XX.X.XX..XX.X...X.X.XX....X...X.X..X.....X........X.] duration:
5.945784355s trace:
https://jaeger.dev.storj.io/trace/1c4c0fba90326f44
FAIL (2025-12-28 05:50:38 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: FBKTDMLHCINVRHMI64CCDXNLA7WWKW2MKGQKK2JNGK4TCJJR4VWA) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: IFFSYXZED5GVA2IKJLCXWZRPNZVYWAN63MGFDPTYSU7D4BMPT3RA) context deadline exceeded
======LOGS======
c 28 05:49:53.964236577 (542h20m53.835319622s) [upload=475605]: Starting upload
Dec 28 05:49:53.964304787 (542h20m53.835387822s) [upload=475605]: Done starting upload
Dec 28 05:49:53.964353770 (542h20m53.835436805s) [upload=475605]: Uploading segments...
Dec 28 05:49:53.964724576 (542h20m53.835807621s) [upload=475605]: Got next segment. Inline: false
Dec 28 05:49:53.964805681 (542h20m53.835888726s) [upload=475605]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 05:49:53.964813275 (542h20m53.83589632s) [upload=475605]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 05:49:54.210544977 (542h20m54.081628022s) [seg_pos={0 0}, upload=475605]: Begin upload segment...
Dec 28 05:49:54.210612576 (542h20m54.081695621s) [seg_pos={0 0}, upload=475605]: Done begin upload segment.
Dec 28 05:49:54.210655959 (542h20m54.081739004s) [noise=true , address=45.91.116.11:4118 , piece_id=FBKTDMLHCINVRHMI64CCDXNLA7WWKW2MKGQKK2JNGK4TCJJR4VWA, seg_pos={0 0}, upload=475605]: Uploading piece...
Dec 28 05:49:54.211437788 (542h20m54.082520833s) [noise=true , address=77.90.31.13:4626 , piece_id=RNJ7BCRCRYKKO2TCJIKEWAANA6PXFHGLYTWOM25X6TQ3LJCJPDMQ, seg_pos={0 0}, upload=475605]: Uploading piece...
Dec 28 05:49:54.212066334 (542h20m54.08314938s) [noise=true , address=77.90.31.18:4805 , piece_id=IFFSYXZED5GVA2IKJLCXWZRPNZVYWAN63MGFDPTYSU7D4BMPT3RA, seg_pos={0 0}, upload=475605]: Uploading piece...
Dec 28 05:49:54.212482277 (542h20m54.083565322s) [noise=true , address=145.79.227.36:4518 , piece_id=NA3L2ESKHDDCJBJHGCJZADLDMTZ5ZZXE2FEZ6GJ4BWEVPRX6U4OQ, seg_pos={0 0}, upload=475605]: Uploading piece...
Dec 28 05:49:54.213019269 (542h20m54.084102314s) [noise=true , address=145.79.227.50:4374 , piece_id=4RKQXOQYWRM7ZC3JQFOILXWWDHPYFZYR3YBJLZRA5RZB2PNVMLKQ, seg_pos={0 0}, upload=475605]: Uploading piece...
Dec 28 05:49:54.213518980 (542h20m54.084602015s) [noise=true , address=145.79.227.51:4411 , piece_id=7Z7WIROUYCJMMISCUSHSN7WCRDSCEWOSLADDDLGK7FZIXHQZE4AA, seg_pos={0 0}, upload=475605]: Uploading piece...
Dec 28 05:49:54.214019624 (542h20m54.085102659s) [noise=true , address=145.79.227.7:4293 , piece_id=VH6QJUZI2P2S3E5CFD3CMY5TOOUFX7P2DOX2LCZT7BHC7GVGKD6Q, seg_pos={0 0}, upload=475605]: Uploading piece...
Dec 28 05:49:54.214565593 (542h20m54.085648638s) [noise=true , address=145.79.227.33:4870 , piece_id=4U6WHLGVALEVHAJOFXG6D7HZ4IU6G5BYOFTGTPEMHQ2G3DK4O2SA, seg_pos={0 0}, upload=475605]: Uploading piece...
Dec 28 05:49:54.216167082 (542h20m54.087250127s) [noise=true , address=45.91.116.13:4155 , piece_id=P3BMQU5EHE7BLVM5XRKDPB2HQIDELVE67QXQ4TWCYSB5DQRVOYOA, seg_pos={0 0}, upload=475605]: Uploading piece...
Dec 28 05:49:54.217939377 (542h20m54.089022432s) [noise=true , address=77.90.31.14:4662 , piece_id=JOCXDUJA3YYWIYVZEOMDAXSX7A7XFAARLYZK5GTE5UKHUN6SKZVA, seg_pos={0 0}, upload=475605]: Uploading piece...
Dec 28 05:49:54.219536669 (542h20m54.090619714s) [noise=true , address=145.79.227.2:4113 , piece_id=H37EOIGKBSNPVYRPD7LZV6AOQPRBREXGDCRZVFZRCFTC4FKAEAQA, seg_pos={0 0}, upload=475605]: Uploading piece...
Dec 28 05:49:54.221295528 (542h20m54.092378573s) [noise=true , address=145.79.227.6:4258 , piece_id=2T2PHFQUXIUYLV54KLS4E5V3FBS7UFGGKJ2AJX7OHVWMGG5US3EQ, seg_pos={0 0}, upload=475605]: Uploading piece...
Dec 28 05:49:54.223541675 (542h20m54.09462471s) [noise=true , address=145.79.227.52:4446 , piece_id=RDJSEQUNF4ESGJMW3OEKEQQF3CCUGRFLOB5TS6T6HF6X4ZFOTOUA, seg_pos={0 0}, upload=475605]: Uploading piece...
Dec 28 05:49:54.225988124 (542h20m54.097071169s) [noise=true , address=145.79.227.48:4303 , piece_id=2C5QMP3KKEGDAEVALCCSJ5KFD4RFY4CNGRVXJMJGNXUI2YYUNXXQ, seg_pos={0 0}, upload=475605]: Uploading piece...
Dec 28 05:49:54.228586531 (542h20m54.099669566s) [noise=true , address=145.79.227.37:4565 , piece_id=ZHTVIULJSWVWH4TKMOGBL3ATODW7OS7JZMYDO5QKZK3Q2AVEWTLQ, seg_pos={0 0}, upload=475605]: Uploading piece...
Dec 28 05:49:54.231333522 (542h20m54.102416567s) [noise=true , address=145.79.227.42:4123 , piece_id=DRQME77DJN6KVBAGGSOZMLGPKGYYTHSFXMXGPSL43IXIGM5AJWBA, seg_pos={0 0}, upload=475605]: Uploading piece...
Dec 28 05:49:55.849579995 (542h20m55.72066303s) [upload=475605]: Next returned nil segment
Dec 28 05:49:55.849594874 (542h20m55.720677909s) [upload=475605]: Waiting for error group managing segments...
Dec 28 05:49:56.246089673 (542h20m56.117172718s) [noise=true , address=145.79.227.42:4123 , piece_id=DRQME77DJN6KVBAGGSOZMLGPKGYYTHSFXMXGPSL43IXIGM5AJWBA, seg_pos={0 0}, upload=475605]: Done uploading piece. err: <nil>
Dec 28 05:49:56.247565743 (542h20m56.118648789s) [noise=true , address=145.79.227.6:4258 , piece_id=2T2PHFQUXIUYLV54KLS4E5V3FBS7UFGGKJ2AJX7OHVWMGG5US3EQ, seg_pos={0 0}, upload=475605]: Done uploading piece. err: <nil>
Dec 28 05:49:56.249641577 (542h20m56.120724622s) [noise=true , address=145.79.227.2:4113 , piece_id=H37EOIGKBSNPVYRPD7LZV6AOQPRBREXGDCRZVFZRCFTC4FKAEAQA, seg_pos={0 0}, upload=475605]: Done uploading piece. err: <nil>
Dec 28 05:49:56.251345281 (542h20m56.122428326s) [noise=true , address=145.79.227.36:4518 , piece_id=NA3L2ESKHDDCJBJHGCJZADLDMTZ5ZZXE2FEZ6GJ4BWEVPRX6U4OQ, seg_pos={0 0}, upload=475605]: Done uploading piece. err: <nil>
Dec 28 05:49:56.254033299 (542h20m56.125116354s) [noise=true , address=145.79.227.50:4374 , piece_id=4RKQXOQYWRM7ZC3JQFOILXWWDHPYFZYR3YBJLZRA5RZB2PNVMLKQ, seg_pos={0 0}, upload=475605]: Done uploading piece. err: <nil>
Dec 28 05:49:56.258547275 (542h20m56.12963032s) [noise=true , address=145.79.227.52:4446 , piece_id=RDJSEQUNF4ESGJMW3OEKEQQF3CCUGRFLOB5TS6T6HF6X4ZFOTOUA, seg_pos={0 0}, upload=475605]: Done uploading piece. err: <nil>
Dec 28 05:49:56.261965754 (542h20m56.133048789s) [noise=true , address=145.79.227.51:4411 , piece_id=7Z7WIROUYCJMMISCUSHSN7WCRDSCEWOSLADDDLGK7FZIXHQZE4AA, seg_pos={0 0}, upload=475605]: Done uploading piece. err: <nil>
Dec 28 05:49:56.262566037 (542h20m56.133649082s) [noise=true , address=145.79.227.48:4303 , piece_id=2C5QMP3KKEGDAEVALCCSJ5KFD4RFY4CNGRVXJMJGNXUI2YYUNXXQ, seg_pos={0 0}, upload=475605]: Done uploading piece. err: <nil>
Dec 28 05:49:56.262842934 (542h20m56.133925979s) [noise=true , address=145.79.227.37:4565 , piece_id=ZHTVIULJSWVWH4TKMOGBL3ATODW7OS7JZMYDO5QKZK3Q2AVEWTLQ, seg_pos={0 0}, upload=475605]: Done uploading piece. err: <nil>
Dec 28 05:49:56.263743830 (542h20m56.134826875s) [noise=true , address=145.79.227.7:4293 , piece_id=VH6QJUZI2P2S3E5CFD3CMY5TOOUFX7P2DOX2LCZT7BHC7GVGKD6Q, seg_pos={0 0}, upload=475605]: Done uploading piece. err: <nil>
Dec 28 05:49:56.521742791 (542h20m56.392825866s) [noise=true , address=45.91.116.11:4118 , piece_id=FBKTDMLHCINVRHMI64CCDXNLA7WWKW2MKGQKK2JNGK4TCJJR4VWA, seg_pos={0 0}, upload=475605]: Done uploading piece. err: <nil>
Dec 28 05:49:56.532519422 (542h20m56.403602457s) [noise=true , address=77.90.31.18:4805 , piece_id=IFFSYXZED5GVA2IKJLCXWZRPNZVYWAN63MGFDPTYSU7D4BMPT3RA, seg_pos={0 0}, upload=475605]: Done uploading piece. err: <nil>
Dec 28 05:49:56.621882173 (542h20m56.492965218s) [noise=true , address=45.91.116.13:4155 , piece_id=P3BMQU5EHE7BLVM5XRKDPB2HQIDELVE67QXQ4TWCYSB5DQRVOYOA, seg_pos={0 0}, upload=475605]: Done uploading piece. err: <nil>
Dec 28 05:49:56.621906769 (542h20m56.492989814s) [seg_pos={0 0}, upload=475605]: Segment reached optimal threshold of 13 pieces.
Dec 28 05:49:56.622133100 (542h20m56.493216135s) [noise=true , address=145.79.227.33:4870 , piece_id=4U6WHLGVALEVHAJOFXG6D7HZ4IU6G5BYOFTGTPEMHQ2G3DK4O2SA, seg_pos={0 0}, upload=475605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF
Dec 28 05:49:56.622143089 (542h20m56.493226124s) [noise=true , address=145.79.227.33:4870 , piece_id=4U6WHLGVALEVHAJOFXG6D7HZ4IU6G5BYOFTGTPEMHQ2G3DK4O2SA, seg_pos={0 0}, upload=475605]: Long tail cancellation due to optimal threshold
Dec 28 05:49:56.622361354 (542h20m56.493444399s) [noise=true , address=77.90.31.14:4662 , piece_id=JOCXDUJA3YYWIYVZEOMDAXSX7A7XFAARLYZK5GTE5UKHUN6SKZVA, seg_pos={0 0}, upload=475605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:55736->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:55736->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55736->77.90.31.14:4662: use of closed network connection
Dec 28 05:49:56.622367477 (542h20m56.493450512s) [noise=true , address=77.90.31.14:4662 , piece_id=JOCXDUJA3YYWIYVZEOMDAXSX7A7XFAARLYZK5GTE5UKHUN6SKZVA, seg_pos={0 0}, upload=475605]: Long tail cancellation due to optimal threshold
Dec 28 05:49:56.622387384 (542h20m56.493470429s) [noise=true , address=77.90.31.13:4626 , piece_id=RNJ7BCRCRYKKO2TCJIKEWAANA6PXFHGLYTWOM25X6TQ3LJCJPDMQ, seg_pos={0 0}, upload=475605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF
Dec 28 05:49:56.622408304 (542h20m56.49349135s) [noise=true , address=77.90.31.13:4626 , piece_id=RNJ7BCRCRYKKO2TCJIKEWAANA6PXFHGLYTWOM25X6TQ3LJCJPDMQ, seg_pos={0 0}, upload=475605]: Long tail cancellation due to optimal threshold
Dec 28 05:49:56.622549654 (542h20m56.493632689s) [seg_pos={0 0}, upload=475605]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 05:49:56.622626440 (542h20m56.493709475s) [upload=475605]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 05:49:56.622631609 (542h20m56.493714654s) [upload=475605]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 05:49:56.919679987 (542h20m56.790763083s) [upload=475605]: Done uploading segments...
Dec 28 05:49:56.919701469 (542h20m56.790784504s) [upload=475605]: Upload finished. err: <nil>
Duration: 5.945806959s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [.......................X.........X...X....................X.] duration:
39.090064561s trace:
https://jaeger.dev.storj.io/trace/35a88c4b3a3d78b6
FAIL (2025-12-28 05:50:30 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH, Piece ID: HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA) context deadline exceeded; piecestore: (Node ID: 1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, Piece ID: GY4LXIIIB7SVWA2WRAEHJVB6HLN6XQYISF7ZPVE4KA42C23MEQ4A) context deadline exceeded; piecestore: (Node ID: 1RQ3vtArvKAvaMJTVa9VDDVdWR5oBuZ7gaPB3pzc1MouqBZhWu, Piece ID: FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ) context deadline exceeded; piecestore: (Node ID: 127KQ2X6VXddEeFRMau7MVGwtTsUmFsm8hTYDR2A6B4A1oVKfyo, Piece ID: V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ) context deadline exceeded
======LOGS======
) [noise=true , address=145.79.227.44:4209 , piece_id=JRNDKXGHFY5OJYSY4ESURYMK5BNRDWY6INLHOWEK55KHMNRJO4AQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.655023976 (542h19m51.526107021s) [noise=true , address=77.90.31.13:4647 , piece_id=SOPU7ZUZQXRLU6TUL4L3ODX7U7WFAKUDOSK5UO66WMAIANU6PVUA, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.655676448 (542h19m51.526759493s) [noise=true , address=45.91.116.8:4059 , piece_id=OSA2YK4AH2DPQBNSJDFT3PLTGPJM6N7EOB7X27ZP2WM52HWSHT4Q, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.656204463 (542h19m51.527287509s) [noise=true , address=145.79.227.49:4370 , piece_id=QDMDBC3576X7ZYGUAC2DLJKWBNVOLSROCMA4UNFDZ47TTADILUAQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.656699927 (542h19m51.527782972s) [noise=true , address=45.91.116.9:4290 , piece_id=MP5DAA5UGHWQ3BA5R7GBWNIV2B4JYCL5WUSV2PY7GTO4PNLH2F2Q, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.657183808 (542h19m51.528266883s) [noise=true , address=77.90.31.18:4827 , piece_id=V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.657682177 (542h19m51.528765222s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.658706768 (542h19m51.529789813s) [noise=true , address=145.79.227.33:4876 , piece_id=E5QQDATADJON5NOOQ6X37GFKUQCFUKJRGK6G2EURBSUJVLF72Y7A, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.660113707 (542h19m51.531196752s) [noise=true , address=145.79.227.34:4915 , piece_id=MR5WOODFPXH57I6O2JCMC3EE5OWXF23CFLLATZXFP5XK6YPWDZJA, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.661356113 (542h19m51.532439158s) [noise=true , address=77.90.31.15:4727 , piece_id=3ACH3RUAR5KUXZU23AL3OQSGIZQOU5MFOFRN3JUYLQJBJCA4JOZQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.664431910 (542h19m51.535514955s) [noise=true , address=45.91.116.6:4998 , piece_id=FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.666377204 (542h19m51.537460249s) [noise=true , address=145.79.227.40:4067 , piece_id=36Q7DHHN5ELVD3BYSNUGJ7WYXNGIDKERTOPRAZJNJ7YQTOUPWEYQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.666754502 (542h19m51.537837548s) [noise=true , address=77.90.31.12:4594 , piece_id=DNXKJQOUHLAWMOCR35THP5CKCB6LZBW4GMJU6O6KKVSIJ3ALLONQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.668451284 (542h19m51.539534319s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.671076552 (542h19m51.542159587s) [noise=true , address=77.90.31.11:4560 , piece_id=HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.674568571 (542h19m51.545651626s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:52.998088476 (542h19m52.869171521s) [upload=475588]: Next returned nil segment
Dec 28 05:48:52.998103846 (542h19m52.869186881s) [upload=475588]: Waiting for error group managing segments...
Dec 28 05:48:53.543654582 (542h19m53.414737627s) [noise=true , address=145.79.227.40:4067 , piece_id=36Q7DHHN5ELVD3BYSNUGJ7WYXNGIDKERTOPRAZJNJ7YQTOUPWEYQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.543744133 (542h19m53.414827178s) [noise=true , address=145.79.227.49:4370 , piece_id=QDMDBC3576X7ZYGUAC2DLJKWBNVOLSROCMA4UNFDZ47TTADILUAQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.556254274 (542h19m53.427337309s) [noise=true , address=145.79.227.34:4915 , piece_id=MR5WOODFPXH57I6O2JCMC3EE5OWXF23CFLLATZXFP5XK6YPWDZJA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.810781113 (542h19m53.681864158s) [noise=true , address=77.90.31.13:4647 , piece_id=SOPU7ZUZQXRLU6TUL4L3ODX7U7WFAKUDOSK5UO66WMAIANU6PVUA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.814762255 (542h19m53.6858453s) [noise=true , address=77.90.31.16:4756 , piece_id=GY4LXIIIB7SVWA2WRAEHJVB6HLN6XQYISF7ZPVE4KA42C23MEQ4A, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.816966662 (542h19m53.688049707s) [noise=true , address=77.90.31.15:4727 , piece_id=3ACH3RUAR5KUXZU23AL3OQSGIZQOU5MFOFRN3JUYLQJBJCA4JOZQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.895077718 (542h19m53.766160783s) [noise=true , address=45.91.116.9:4290 , piece_id=MP5DAA5UGHWQ3BA5R7GBWNIV2B4JYCL5WUSV2PY7GTO4PNLH2F2Q, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.906586933 (542h19m53.777669978s) [noise=true , address=45.91.116.8:4059 , piece_id=OSA2YK4AH2DPQBNSJDFT3PLTGPJM6N7EOB7X27ZP2WM52HWSHT4Q, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.930785842 (542h19m53.801868877s) [noise=true , address=145.79.227.33:4876 , piece_id=E5QQDATADJON5NOOQ6X37GFKUQCFUKJRGK6G2EURBSUJVLF72Y7A, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.087446700 (542h19m53.958529735s) [noise=true , address=45.91.116.6:4998 , piece_id=FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.129981903 (542h19m54.001064938s) [noise=true , address=77.90.31.11:4560 , piece_id=HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.264740966 (542h19m54.135824011s) [noise=true , address=77.90.31.12:4594 , piece_id=DNXKJQOUHLAWMOCR35THP5CKCB6LZBW4GMJU6O6KKVSIJ3ALLONQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.368348401 (542h19m54.239431446s) [noise=true , address=77.90.31.18:4827 , piece_id=V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.437713885 (542h19m54.30879693s) [noise=true , address=145.79.227.44:4209 , piece_id=JRNDKXGHFY5OJYSY4ESURYMK5BNRDWY6INLHOWEK55KHMNRJO4AQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.437828132 (542h19m54.308911187s) [seg_pos={0 0}, upload=475588]: Segment reached optimal threshold of 14 pieces.
Dec 28 05:48:54.438064122 (542h19m54.309147167s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF
Dec 28 05:48:54.438074481 (542h19m54.309157546s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold
Dec 28 05:48:54.438099619 (542h19m54.309182664s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F4fJAGJM2ZPRt7vJH2cZmEbYeujmxcfi7aPZ2g9VPTNYBYRAi): protocol: expected piece hash; context canceled; EOF
Dec 28 05:48:54.438108586 (542h19m54.309191631s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold
Dec 28 05:48:54.438243403 (542h19m54.309326458s) [noise=true , address=77.90.31.19:4876 , piece_id=PUIQX7IZSNPKZTZTYH32KTFUM7K53KWZASLPT6JHAPMOB2JCXPGA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12H3UNBGeAufYSdVvEgYkv2QWeFtjKMvHvYGeqGeBvjEHqDDjBB): protocol: expected piece hash; context canceled; EOF
Dec 28 05:48:54.438252169 (542h19m54.309335214s) [noise=true , address=77.90.31.19:4876 , piece_id=PUIQX7IZSNPKZTZTYH32KTFUM7K53KWZASLPT6JHAPMOB2JCXPGA, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold
Dec 28 05:48:54.438391024 (542h19m54.309474069s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection
Dec 28 05:48:54.438398588 (542h19m54.309481633s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold
Dec 28 05:48:54.438699081 (542h19m54.309782126s) [seg_pos={0 0}, upload=475588]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 05:48:54.438794753 (542h19m54.309877798s) [upload=475588]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 05:48:54.438801255 (542h19m54.30988429s) [upload=475588]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 05:48:54.728435695 (542h19m54.59951876s) [upload=475588]: Done uploading segments...
Dec 28 05:48:54.728457938 (542h19m54.599540983s) [upload=475588]: Upload finished. err: <nil>
Duration: 39.090083107s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
20 out of 60 is failed (threshold=3) [...X...X.XX.X.XX..XX.X...X.X.XX....X...X.X..X.....X........X] duration:
30.003429731s trace:
https://jaeger.dev.storj.io/trace/1c4c0fba90326f44
FAIL (2025-12-28 05:50:23 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: FBKTDMLHCINVRHMI64CCDXNLA7WWKW2MKGQKK2JNGK4TCJJR4VWA) context deadline exceeded; piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: IFFSYXZED5GVA2IKJLCXWZRPNZVYWAN63MGFDPTYSU7D4BMPT3RA) context deadline exceeded
======LOGS======
c 28 05:49:53.964236577 (542h20m53.835319622s) [upload=475605]: Starting upload
Dec 28 05:49:53.964304787 (542h20m53.835387822s) [upload=475605]: Done starting upload
Dec 28 05:49:53.964353770 (542h20m53.835436805s) [upload=475605]: Uploading segments...
Dec 28 05:49:53.964724576 (542h20m53.835807621s) [upload=475605]: Got next segment. Inline: false
Dec 28 05:49:53.964805681 (542h20m53.835888726s) [upload=475605]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 05:49:53.964813275 (542h20m53.83589632s) [upload=475605]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 05:49:54.210544977 (542h20m54.081628022s) [seg_pos={0 0}, upload=475605]: Begin upload segment...
Dec 28 05:49:54.210612576 (542h20m54.081695621s) [seg_pos={0 0}, upload=475605]: Done begin upload segment.
Dec 28 05:49:54.210655959 (542h20m54.081739004s) [noise=true , address=45.91.116.11:4118 , piece_id=FBKTDMLHCINVRHMI64CCDXNLA7WWKW2MKGQKK2JNGK4TCJJR4VWA, seg_pos={0 0}, upload=475605]: Uploading piece...
Dec 28 05:49:54.211437788 (542h20m54.082520833s) [noise=true , address=77.90.31.13:4626 , piece_id=RNJ7BCRCRYKKO2TCJIKEWAANA6PXFHGLYTWOM25X6TQ3LJCJPDMQ, seg_pos={0 0}, upload=475605]: Uploading piece...
Dec 28 05:49:54.212066334 (542h20m54.08314938s) [noise=true , address=77.90.31.18:4805 , piece_id=IFFSYXZED5GVA2IKJLCXWZRPNZVYWAN63MGFDPTYSU7D4BMPT3RA, seg_pos={0 0}, upload=475605]: Uploading piece...
Dec 28 05:49:54.212482277 (542h20m54.083565322s) [noise=true , address=145.79.227.36:4518 , piece_id=NA3L2ESKHDDCJBJHGCJZADLDMTZ5ZZXE2FEZ6GJ4BWEVPRX6U4OQ, seg_pos={0 0}, upload=475605]: Uploading piece...
Dec 28 05:49:54.213019269 (542h20m54.084102314s) [noise=true , address=145.79.227.50:4374 , piece_id=4RKQXOQYWRM7ZC3JQFOILXWWDHPYFZYR3YBJLZRA5RZB2PNVMLKQ, seg_pos={0 0}, upload=475605]: Uploading piece...
Dec 28 05:49:54.213518980 (542h20m54.084602015s) [noise=true , address=145.79.227.51:4411 , piece_id=7Z7WIROUYCJMMISCUSHSN7WCRDSCEWOSLADDDLGK7FZIXHQZE4AA, seg_pos={0 0}, upload=475605]: Uploading piece...
Dec 28 05:49:54.214019624 (542h20m54.085102659s) [noise=true , address=145.79.227.7:4293 , piece_id=VH6QJUZI2P2S3E5CFD3CMY5TOOUFX7P2DOX2LCZT7BHC7GVGKD6Q, seg_pos={0 0}, upload=475605]: Uploading piece...
Dec 28 05:49:54.214565593 (542h20m54.085648638s) [noise=true , address=145.79.227.33:4870 , piece_id=4U6WHLGVALEVHAJOFXG6D7HZ4IU6G5BYOFTGTPEMHQ2G3DK4O2SA, seg_pos={0 0}, upload=475605]: Uploading piece...
Dec 28 05:49:54.216167082 (542h20m54.087250127s) [noise=true , address=45.91.116.13:4155 , piece_id=P3BMQU5EHE7BLVM5XRKDPB2HQIDELVE67QXQ4TWCYSB5DQRVOYOA, seg_pos={0 0}, upload=475605]: Uploading piece...
Dec 28 05:49:54.217939377 (542h20m54.089022432s) [noise=true , address=77.90.31.14:4662 , piece_id=JOCXDUJA3YYWIYVZEOMDAXSX7A7XFAARLYZK5GTE5UKHUN6SKZVA, seg_pos={0 0}, upload=475605]: Uploading piece...
Dec 28 05:49:54.219536669 (542h20m54.090619714s) [noise=true , address=145.79.227.2:4113 , piece_id=H37EOIGKBSNPVYRPD7LZV6AOQPRBREXGDCRZVFZRCFTC4FKAEAQA, seg_pos={0 0}, upload=475605]: Uploading piece...
Dec 28 05:49:54.221295528 (542h20m54.092378573s) [noise=true , address=145.79.227.6:4258 , piece_id=2T2PHFQUXIUYLV54KLS4E5V3FBS7UFGGKJ2AJX7OHVWMGG5US3EQ, seg_pos={0 0}, upload=475605]: Uploading piece...
Dec 28 05:49:54.223541675 (542h20m54.09462471s) [noise=true , address=145.79.227.52:4446 , piece_id=RDJSEQUNF4ESGJMW3OEKEQQF3CCUGRFLOB5TS6T6HF6X4ZFOTOUA, seg_pos={0 0}, upload=475605]: Uploading piece...
Dec 28 05:49:54.225988124 (542h20m54.097071169s) [noise=true , address=145.79.227.48:4303 , piece_id=2C5QMP3KKEGDAEVALCCSJ5KFD4RFY4CNGRVXJMJGNXUI2YYUNXXQ, seg_pos={0 0}, upload=475605]: Uploading piece...
Dec 28 05:49:54.228586531 (542h20m54.099669566s) [noise=true , address=145.79.227.37:4565 , piece_id=ZHTVIULJSWVWH4TKMOGBL3ATODW7OS7JZMYDO5QKZK3Q2AVEWTLQ, seg_pos={0 0}, upload=475605]: Uploading piece...
Dec 28 05:49:54.231333522 (542h20m54.102416567s) [noise=true , address=145.79.227.42:4123 , piece_id=DRQME77DJN6KVBAGGSOZMLGPKGYYTHSFXMXGPSL43IXIGM5AJWBA, seg_pos={0 0}, upload=475605]: Uploading piece...
Dec 28 05:49:55.849579995 (542h20m55.72066303s) [upload=475605]: Next returned nil segment
Dec 28 05:49:55.849594874 (542h20m55.720677909s) [upload=475605]: Waiting for error group managing segments...
Dec 28 05:49:56.246089673 (542h20m56.117172718s) [noise=true , address=145.79.227.42:4123 , piece_id=DRQME77DJN6KVBAGGSOZMLGPKGYYTHSFXMXGPSL43IXIGM5AJWBA, seg_pos={0 0}, upload=475605]: Done uploading piece. err: <nil>
Dec 28 05:49:56.247565743 (542h20m56.118648789s) [noise=true , address=145.79.227.6:4258 , piece_id=2T2PHFQUXIUYLV54KLS4E5V3FBS7UFGGKJ2AJX7OHVWMGG5US3EQ, seg_pos={0 0}, upload=475605]: Done uploading piece. err: <nil>
Dec 28 05:49:56.249641577 (542h20m56.120724622s) [noise=true , address=145.79.227.2:4113 , piece_id=H37EOIGKBSNPVYRPD7LZV6AOQPRBREXGDCRZVFZRCFTC4FKAEAQA, seg_pos={0 0}, upload=475605]: Done uploading piece. err: <nil>
Dec 28 05:49:56.251345281 (542h20m56.122428326s) [noise=true , address=145.79.227.36:4518 , piece_id=NA3L2ESKHDDCJBJHGCJZADLDMTZ5ZZXE2FEZ6GJ4BWEVPRX6U4OQ, seg_pos={0 0}, upload=475605]: Done uploading piece. err: <nil>
Dec 28 05:49:56.254033299 (542h20m56.125116354s) [noise=true , address=145.79.227.50:4374 , piece_id=4RKQXOQYWRM7ZC3JQFOILXWWDHPYFZYR3YBJLZRA5RZB2PNVMLKQ, seg_pos={0 0}, upload=475605]: Done uploading piece. err: <nil>
Dec 28 05:49:56.258547275 (542h20m56.12963032s) [noise=true , address=145.79.227.52:4446 , piece_id=RDJSEQUNF4ESGJMW3OEKEQQF3CCUGRFLOB5TS6T6HF6X4ZFOTOUA, seg_pos={0 0}, upload=475605]: Done uploading piece. err: <nil>
Dec 28 05:49:56.261965754 (542h20m56.133048789s) [noise=true , address=145.79.227.51:4411 , piece_id=7Z7WIROUYCJMMISCUSHSN7WCRDSCEWOSLADDDLGK7FZIXHQZE4AA, seg_pos={0 0}, upload=475605]: Done uploading piece. err: <nil>
Dec 28 05:49:56.262566037 (542h20m56.133649082s) [noise=true , address=145.79.227.48:4303 , piece_id=2C5QMP3KKEGDAEVALCCSJ5KFD4RFY4CNGRVXJMJGNXUI2YYUNXXQ, seg_pos={0 0}, upload=475605]: Done uploading piece. err: <nil>
Dec 28 05:49:56.262842934 (542h20m56.133925979s) [noise=true , address=145.79.227.37:4565 , piece_id=ZHTVIULJSWVWH4TKMOGBL3ATODW7OS7JZMYDO5QKZK3Q2AVEWTLQ, seg_pos={0 0}, upload=475605]: Done uploading piece. err: <nil>
Dec 28 05:49:56.263743830 (542h20m56.134826875s) [noise=true , address=145.79.227.7:4293 , piece_id=VH6QJUZI2P2S3E5CFD3CMY5TOOUFX7P2DOX2LCZT7BHC7GVGKD6Q, seg_pos={0 0}, upload=475605]: Done uploading piece. err: <nil>
Dec 28 05:49:56.521742791 (542h20m56.392825866s) [noise=true , address=45.91.116.11:4118 , piece_id=FBKTDMLHCINVRHMI64CCDXNLA7WWKW2MKGQKK2JNGK4TCJJR4VWA, seg_pos={0 0}, upload=475605]: Done uploading piece. err: <nil>
Dec 28 05:49:56.532519422 (542h20m56.403602457s) [noise=true , address=77.90.31.18:4805 , piece_id=IFFSYXZED5GVA2IKJLCXWZRPNZVYWAN63MGFDPTYSU7D4BMPT3RA, seg_pos={0 0}, upload=475605]: Done uploading piece. err: <nil>
Dec 28 05:49:56.621882173 (542h20m56.492965218s) [noise=true , address=45.91.116.13:4155 , piece_id=P3BMQU5EHE7BLVM5XRKDPB2HQIDELVE67QXQ4TWCYSB5DQRVOYOA, seg_pos={0 0}, upload=475605]: Done uploading piece. err: <nil>
Dec 28 05:49:56.621906769 (542h20m56.492989814s) [seg_pos={0 0}, upload=475605]: Segment reached optimal threshold of 13 pieces.
Dec 28 05:49:56.622133100 (542h20m56.493216135s) [noise=true , address=145.79.227.33:4870 , piece_id=4U6WHLGVALEVHAJOFXG6D7HZ4IU6G5BYOFTGTPEMHQ2G3DK4O2SA, seg_pos={0 0}, upload=475605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF
Dec 28 05:49:56.622143089 (542h20m56.493226124s) [noise=true , address=145.79.227.33:4870 , piece_id=4U6WHLGVALEVHAJOFXG6D7HZ4IU6G5BYOFTGTPEMHQ2G3DK4O2SA, seg_pos={0 0}, upload=475605]: Long tail cancellation due to optimal threshold
Dec 28 05:49:56.622361354 (542h20m56.493444399s) [noise=true , address=77.90.31.14:4662 , piece_id=JOCXDUJA3YYWIYVZEOMDAXSX7A7XFAARLYZK5GTE5UKHUN6SKZVA, seg_pos={0 0}, upload=475605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n): protocol: write tcp 188.166.225.170:55736->77.90.31.14:4662: use of closed network connection; write tcp 188.166.225.170:55736->77.90.31.14:4662: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:55736->77.90.31.14:4662: use of closed network connection
Dec 28 05:49:56.622367477 (542h20m56.493450512s) [noise=true , address=77.90.31.14:4662 , piece_id=JOCXDUJA3YYWIYVZEOMDAXSX7A7XFAARLYZK5GTE5UKHUN6SKZVA, seg_pos={0 0}, upload=475605]: Long tail cancellation due to optimal threshold
Dec 28 05:49:56.622387384 (542h20m56.493470429s) [noise=true , address=77.90.31.13:4626 , piece_id=RNJ7BCRCRYKKO2TCJIKEWAANA6PXFHGLYTWOM25X6TQ3LJCJPDMQ, seg_pos={0 0}, upload=475605]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF
Dec 28 05:49:56.622408304 (542h20m56.49349135s) [noise=true , address=77.90.31.13:4626 , piece_id=RNJ7BCRCRYKKO2TCJIKEWAANA6PXFHGLYTWOM25X6TQ3LJCJPDMQ, seg_pos={0 0}, upload=475605]: Long tail cancellation due to optimal threshold
Dec 28 05:49:56.622549654 (542h20m56.493632689s) [seg_pos={0 0}, upload=475605]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 05:49:56.622626440 (542h20m56.493709475s) [upload=475605]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 05:49:56.622631609 (542h20m56.493714654s) [upload=475605]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 05:49:56.919679987 (542h20m56.790763083s) [upload=475605]: Done uploading segments...
Dec 28 05:49:56.919701469 (542h20m56.790784504s) [upload=475605]: Upload finished. err: <nil>
Duration: 30.003456371s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
4 out of 60 is failed (threshold=3) [........................X.........X...X....................X] duration:
1m0.00381415s trace:
https://jaeger.dev.storj.io/trace/35a88c4b3a3d78b6
FAIL (2025-12-28 05:49:50 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 1YWmyb9dcfFj82ShgsB6EnQXm3mx1YSLvWc6P1SZhREksqXUUH, Piece ID: HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA) context deadline exceeded; piecestore: (Node ID: 1UaCjYcwJTUUrVNVJmJL4m2Gpcw9gcyGLDHySu1e3hmSkXYbtX, Piece ID: GY4LXIIIB7SVWA2WRAEHJVB6HLN6XQYISF7ZPVE4KA42C23MEQ4A) context deadline exceeded; piecestore: (Node ID: 1RQ3vtArvKAvaMJTVa9VDDVdWR5oBuZ7gaPB3pzc1MouqBZhWu, Piece ID: FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ) context deadline exceeded; piecestore: (Node ID: 127KQ2X6VXddEeFRMau7MVGwtTsUmFsm8hTYDR2A6B4A1oVKfyo, Piece ID: V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ) context deadline exceeded
======LOGS======
) [noise=true , address=145.79.227.44:4209 , piece_id=JRNDKXGHFY5OJYSY4ESURYMK5BNRDWY6INLHOWEK55KHMNRJO4AQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.655023976 (542h19m51.526107021s) [noise=true , address=77.90.31.13:4647 , piece_id=SOPU7ZUZQXRLU6TUL4L3ODX7U7WFAKUDOSK5UO66WMAIANU6PVUA, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.655676448 (542h19m51.526759493s) [noise=true , address=45.91.116.8:4059 , piece_id=OSA2YK4AH2DPQBNSJDFT3PLTGPJM6N7EOB7X27ZP2WM52HWSHT4Q, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.656204463 (542h19m51.527287509s) [noise=true , address=145.79.227.49:4370 , piece_id=QDMDBC3576X7ZYGUAC2DLJKWBNVOLSROCMA4UNFDZ47TTADILUAQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.656699927 (542h19m51.527782972s) [noise=true , address=45.91.116.9:4290 , piece_id=MP5DAA5UGHWQ3BA5R7GBWNIV2B4JYCL5WUSV2PY7GTO4PNLH2F2Q, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.657183808 (542h19m51.528266883s) [noise=true , address=77.90.31.18:4827 , piece_id=V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.657682177 (542h19m51.528765222s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.658706768 (542h19m51.529789813s) [noise=true , address=145.79.227.33:4876 , piece_id=E5QQDATADJON5NOOQ6X37GFKUQCFUKJRGK6G2EURBSUJVLF72Y7A, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.660113707 (542h19m51.531196752s) [noise=true , address=145.79.227.34:4915 , piece_id=MR5WOODFPXH57I6O2JCMC3EE5OWXF23CFLLATZXFP5XK6YPWDZJA, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.661356113 (542h19m51.532439158s) [noise=true , address=77.90.31.15:4727 , piece_id=3ACH3RUAR5KUXZU23AL3OQSGIZQOU5MFOFRN3JUYLQJBJCA4JOZQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.664431910 (542h19m51.535514955s) [noise=true , address=45.91.116.6:4998 , piece_id=FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.666377204 (542h19m51.537460249s) [noise=true , address=145.79.227.40:4067 , piece_id=36Q7DHHN5ELVD3BYSNUGJ7WYXNGIDKERTOPRAZJNJ7YQTOUPWEYQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.666754502 (542h19m51.537837548s) [noise=true , address=77.90.31.12:4594 , piece_id=DNXKJQOUHLAWMOCR35THP5CKCB6LZBW4GMJU6O6KKVSIJ3ALLONQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.668451284 (542h19m51.539534319s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.671076552 (542h19m51.542159587s) [noise=true , address=77.90.31.11:4560 , piece_id=HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:51.674568571 (542h19m51.545651626s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Uploading piece...
Dec 28 05:48:52.998088476 (542h19m52.869171521s) [upload=475588]: Next returned nil segment
Dec 28 05:48:52.998103846 (542h19m52.869186881s) [upload=475588]: Waiting for error group managing segments...
Dec 28 05:48:53.543654582 (542h19m53.414737627s) [noise=true , address=145.79.227.40:4067 , piece_id=36Q7DHHN5ELVD3BYSNUGJ7WYXNGIDKERTOPRAZJNJ7YQTOUPWEYQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.543744133 (542h19m53.414827178s) [noise=true , address=145.79.227.49:4370 , piece_id=QDMDBC3576X7ZYGUAC2DLJKWBNVOLSROCMA4UNFDZ47TTADILUAQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.556254274 (542h19m53.427337309s) [noise=true , address=145.79.227.34:4915 , piece_id=MR5WOODFPXH57I6O2JCMC3EE5OWXF23CFLLATZXFP5XK6YPWDZJA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.810781113 (542h19m53.681864158s) [noise=true , address=77.90.31.13:4647 , piece_id=SOPU7ZUZQXRLU6TUL4L3ODX7U7WFAKUDOSK5UO66WMAIANU6PVUA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.814762255 (542h19m53.6858453s) [noise=true , address=77.90.31.16:4756 , piece_id=GY4LXIIIB7SVWA2WRAEHJVB6HLN6XQYISF7ZPVE4KA42C23MEQ4A, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.816966662 (542h19m53.688049707s) [noise=true , address=77.90.31.15:4727 , piece_id=3ACH3RUAR5KUXZU23AL3OQSGIZQOU5MFOFRN3JUYLQJBJCA4JOZQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.895077718 (542h19m53.766160783s) [noise=true , address=45.91.116.9:4290 , piece_id=MP5DAA5UGHWQ3BA5R7GBWNIV2B4JYCL5WUSV2PY7GTO4PNLH2F2Q, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.906586933 (542h19m53.777669978s) [noise=true , address=45.91.116.8:4059 , piece_id=OSA2YK4AH2DPQBNSJDFT3PLTGPJM6N7EOB7X27ZP2WM52HWSHT4Q, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:53.930785842 (542h19m53.801868877s) [noise=true , address=145.79.227.33:4876 , piece_id=E5QQDATADJON5NOOQ6X37GFKUQCFUKJRGK6G2EURBSUJVLF72Y7A, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.087446700 (542h19m53.958529735s) [noise=true , address=45.91.116.6:4998 , piece_id=FRIZZ7SALO24NYRVKJQTXMBNMJ2SVXWNQ6RP36T3OIBD6JYVAXAQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.129981903 (542h19m54.001064938s) [noise=true , address=77.90.31.11:4560 , piece_id=HNUB7ZBNTUS4Z5ZHFPGZF4T4Y22FBZULVNKHT6TS3ZNW4JZAGVBA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.264740966 (542h19m54.135824011s) [noise=true , address=77.90.31.12:4594 , piece_id=DNXKJQOUHLAWMOCR35THP5CKCB6LZBW4GMJU6O6KKVSIJ3ALLONQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.368348401 (542h19m54.239431446s) [noise=true , address=77.90.31.18:4827 , piece_id=V5IMSRVCRXHZNTGHX2T4ODTEFR5MDCRJTRA5FIWP67DXTSJ2FJQQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.437713885 (542h19m54.30879693s) [noise=true , address=145.79.227.44:4209 , piece_id=JRNDKXGHFY5OJYSY4ESURYMK5BNRDWY6INLHOWEK55KHMNRJO4AQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: <nil>
Dec 28 05:48:54.437828132 (542h19m54.308911187s) [seg_pos={0 0}, upload=475588]: Segment reached optimal threshold of 14 pieces.
Dec 28 05:48:54.438064122 (542h19m54.309147167s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12JDCvu6aakv5c9SmN85pAGxBp4xWEuEAyWv5y5Fd5gexm7UidV): protocol: expected piece hash; context canceled; EOF
Dec 28 05:48:54.438074481 (542h19m54.309157546s) [noise=true , address=145.79.227.30:4778 , piece_id=QSHJNERQ4HMRQXK2JKQDN72L2H6CFV3ASRKAJWYLZISM65V5QIIQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold
Dec 28 05:48:54.438099619 (542h19m54.309182664s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12F4fJAGJM2ZPRt7vJH2cZmEbYeujmxcfi7aPZ2g9VPTNYBYRAi): protocol: expected piece hash; context canceled; EOF
Dec 28 05:48:54.438108586 (542h19m54.309191631s) [noise=true , address=193.149.164.143:4698 , piece_id=5IG4NLSWYWD2VHZ36GJGRO53WUG6VFIVMLRGVRLEY2IM2OJ3JYTQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold
Dec 28 05:48:54.438243403 (542h19m54.309326458s) [noise=true , address=77.90.31.19:4876 , piece_id=PUIQX7IZSNPKZTZTYH32KTFUM7K53KWZASLPT6JHAPMOB2JCXPGA, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12H3UNBGeAufYSdVvEgYkv2QWeFtjKMvHvYGeqGeBvjEHqDDjBB): protocol: expected piece hash; context canceled; EOF
Dec 28 05:48:54.438252169 (542h19m54.309335214s) [noise=true , address=77.90.31.19:4876 , piece_id=PUIQX7IZSNPKZTZTYH32KTFUM7K53KWZASLPT6JHAPMOB2JCXPGA, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold
Dec 28 05:48:54.438391024 (542h19m54.309474069s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1Uay43uEvzmHudZaGM1ZcEFbEdUXzQto5RtZNiQXgrqaRmuQoy): protocol: write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection; write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52930->77.90.31.10:4549: use of closed network connection
Dec 28 05:48:54.438398588 (542h19m54.309481633s) [noise=true , address=77.90.31.10:4549 , piece_id=EAH6TJDDE477IXBYWGYACLZJQ7FDCXNCVAQNRAZ5X6OYXXMYXPBQ, seg_pos={0 0}, upload=475588]: Long tail cancellation due to optimal threshold
Dec 28 05:48:54.438699081 (542h19m54.309782126s) [seg_pos={0 0}, upload=475588]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 05:48:54.438794753 (542h19m54.309877798s) [upload=475588]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 05:48:54.438801255 (542h19m54.30988429s) [upload=475588]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 05:48:54.728435695 (542h19m54.59951876s) [upload=475588]: Done uploading segments...
Dec 28 05:48:54.728457938 (542h19m54.599540983s) [upload=475588]: Upload finished. err: <nil>
Duration: 1m0.003829141s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
26 out of 60 is failed (threshold=6) [X.XX.XXX.....XXX....XXXX....XXX.X...X....X...XX.X..XXX....X.] duration:
3.619241408s trace:
https://jaeger.dev.storj.io/trace/4fb94e1d2f36d9ac
FAIL (2025-12-28 05:49:38 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.619266285s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
26 out of 60 is failed (threshold=6) [.X.XX.XXX.....XXX....XXXX....XXX.X...X....X...XX.X..XXX....X] duration:
5.001704458s trace:
https://jaeger.dev.storj.io/trace/4fb94e1d2f36d9ac
FAIL (2025-12-28 05:48:51 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.0017216s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
20 out of 60 is failed (threshold=3) [X...X...X.XX.X.XX..XX.X...X.X.XX....X...X.X..X.....X........] duration:
5.98498388s trace:
https://jaeger.dev.storj.io/trace/58bbbbe9bef3e1af
FAIL (2025-12-28 05:48:50 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 28 05:42:53.387982259 (542h13m53.259065304s) [upload=475503]: Starting upload
Dec 28 05:42:53.388039338 (542h13m53.259122384s) [upload=475503]: Done starting upload
Dec 28 05:42:53.388153736 (542h13m53.259236781s) [upload=475503]: Uploading segments...
Dec 28 05:42:53.388318510 (542h13m53.259401555s) [upload=475503]: Got next segment. Inline: false
Dec 28 05:42:53.388357434 (542h13m53.259440469s) [upload=475503]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 05:42:53.388363225 (542h13m53.25944627s) [upload=475503]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 05:42:53.640155558 (542h13m53.511238603s) [seg_pos={0 0}, upload=475503]: Begin upload segment...
Dec 28 05:42:53.640310503 (542h13m53.511393548s) [seg_pos={0 0}, upload=475503]: Done begin upload segment.
Dec 28 05:42:53.640394433 (542h13m53.511477468s) [noise=true , address=145.79.227.2:4115 , piece_id=AALV2IIJHUEG4GTR5OQFTRN2F4DT5WYUOXELRTUU3NWC6ETFLFWQ, seg_pos={0 0}, upload=475503]: Uploading piece...
Dec 28 05:42:53.640435691 (542h13m53.511518726s) [noise=true , address=45.91.116.9:4262 , piece_id=NCG3ZPSRM4HGNN4L4WCG5V7MOGERCM7H37NAR6ECJVIJNJIUIBNA, seg_pos={0 0}, upload=475503]: Uploading piece...
Dec 28 05:42:53.641347858 (542h13m53.512430893s) [noise=true , address=145.79.227.37:4567 , piece_id=E2KSUSOQMK37YVP3G6FKFJ64RJXRMP7Q72HTEUVVJOVUSB33TRQA, seg_pos={0 0}, upload=475503]: Uploading piece...
Dec 28 05:42:53.642303769 (542h13m53.513386804s) [noise=true , address=145.79.227.31:4798 , piece_id=NXUROFV5C47O2LJFARP5FVO2B765MZKE6UNUMSEJIMBESKBQT6MA, seg_pos={0 0}, upload=475503]: Uploading piece...
Dec 28 05:42:53.643155851 (542h13m53.514238896s) [noise=true , address=145.79.227.15:4876 , piece_id=BMRS3CONOVJ6YTBJR735AALCRPTFV7G7S46MHWZ2RKX2KH3ZHHXA, seg_pos={0 0}, upload=475503]: Uploading piece...
Dec 28 05:42:53.644484592 (542h13m53.515567627s) [noise=true , address=145.79.227.36:4518 , piece_id=NRKTGUWXYI3D6SQ4PJRTY6FATLBFUU7LRFXYDJJMB5FLFDYVE32A, seg_pos={0 0}, upload=475503]: Uploading piece...
Dec 28 05:42:53.646190490 (542h13m53.517273535s) [noise=true , address=45.91.116.6:4975 , piece_id=HJCXHJNNXEUXGUCTYOJ7OMQDNDLSJYDHQP4KS2KHSW4KF3YBU3QQ, seg_pos={0 0}, upload=475503]: Uploading piece...
Dec 28 05:42:53.648465171 (542h13m53.519548216s) [noise=true , address=145.79.227.52:4445 , piece_id=A5TCQ2GV5Q7P6SUF6PK4BHF7SAYGOTAFENV5GAQ4I6NDFNWD2OBQ, seg_pos={0 0}, upload=475503]: Uploading piece...
Dec 28 05:42:53.651026908 (542h13m53.522109954s) [noise=true , address=145.79.227.11:4731 , piece_id=R3U3KCW5AXIUHODBCWZQMMERIM3NRKEQ4CFOBGLVFSS3QL3K2DLA, seg_pos={0 0}, upload=475503]: Uploading piece...
Dec 28 05:42:53.651705571 (542h13m53.522788606s) [noise=true , address=145.79.227.46:4603 , piece_id=BSK5JGLNRAOAEJ6FPYDE46JN5GSF7THRRUAY7YUAUI5NYCQTXLIQ, seg_pos={0 0}, upload=475503]: Uploading piece...
Dec 28 05:42:53.653001268 (542h13m53.524084303s) [noise=true , address=45.91.116.14:4191 , piece_id=44LJ44JTITXMGACUSBV56ND5DGWA3GD4WNQB35K2OSZXOSYRPUJQ, seg_pos={0 0}, upload=475503]: Uploading piece...
Dec 28 05:42:53.654305932 (542h13m53.525388977s) [noise=true , address=145.79.227.50:4374 , piece_id=5DIZD6MCNYMVTBWP6CPB74NVSQB72J6MVTW36EG6QQEO5J4OO3MA, seg_pos={0 0}, upload=475503]: Uploading piece...
Dec 28 05:42:53.656135045 (542h13m53.527218091s) [noise=true , address=145.79.227.32:4835 , piece_id=37M4346GVUKBGONLZ662ITUZIA65AIFEXL5KGONXX3EEP4JSWO3A, seg_pos={0 0}, upload=475503]: Uploading piece...
Dec 28 05:42:53.657413991 (542h13m53.528497036s) [noise=true , address=45.91.116.5:4299 , piece_id=BAVGDZER3LGWNTG47FDWJEA4MO57T6D3STMNPML75KXSNJOEJYEA, seg_pos={0 0}, upload=475503]: Uploading piece...
Dec 28 05:42:53.657754859 (542h13m53.528837904s) [noise=true , address=45.91.116.12:4225 , piece_id=TTULQP7LJP2XPJHG4GK7SM7TA3RP7IACVVWQXQGSFK2W2RIL5TNQ, seg_pos={0 0}, upload=475503]: Uploading piece...
Dec 28 05:42:53.659438286 (542h13m53.530521331s) [noise=true , address=145.79.227.17:4970 , piece_id=CSPT4WC7HLSBBFMOMWKU23XPHUFQCGLFLSFMZTQWMNUA4D4QNY4Q, seg_pos={0 0}, upload=475503]: Uploading piece...
Dec 28 05:42:55.360476706 (542h13m55.231559751s) [upload=475503]: Next returned nil segment
Dec 28 05:42:55.360555025 (542h13m55.23163807s) [upload=475503]: Waiting for error group managing segments...
Dec 28 05:42:55.652562073 (542h13m55.523645118s) [noise=true , address=145.79.227.32:4835 , piece_id=37M4346GVUKBGONLZ662ITUZIA65AIFEXL5KGONXX3EEP4JSWO3A, seg_pos={0 0}, upload=475503]: Done uploading piece. err: <nil>
Dec 28 05:42:55.652599064 (542h13m55.523682129s) [noise=true , address=145.79.227.2:4115 , piece_id=AALV2IIJHUEG4GTR5OQFTRN2F4DT5WYUOXELRTUU3NWC6ETFLFWQ, seg_pos={0 0}, upload=475503]: Done uploading piece. err: <nil>
Dec 28 05:42:55.657515976 (542h13m55.528599021s) [noise=true , address=145.79.227.52:4445 , piece_id=A5TCQ2GV5Q7P6SUF6PK4BHF7SAYGOTAFENV5GAQ4I6NDFNWD2OBQ, seg_pos={0 0}, upload=475503]: Done uploading piece. err: <nil>
Dec 28 05:42:55.657674639 (542h13m55.528757684s) [noise=true , address=145.79.227.31:4798 , piece_id=NXUROFV5C47O2LJFARP5FVO2B765MZKE6UNUMSEJIMBESKBQT6MA, seg_pos={0 0}, upload=475503]: Done uploading piece. err: <nil>
Dec 28 05:42:55.658848644 (542h13m55.52993168s) [noise=true , address=145.79.227.37:4567 , piece_id=E2KSUSOQMK37YVP3G6FKFJ64RJXRMP7Q72HTEUVVJOVUSB33TRQA, seg_pos={0 0}, upload=475503]: Done uploading piece. err: <nil>
Dec 28 05:42:55.663002935 (542h13m55.53408597s) [noise=true , address=145.79.227.50:4374 , piece_id=5DIZD6MCNYMVTBWP6CPB74NVSQB72J6MVTW36EG6QQEO5J4OO3MA, seg_pos={0 0}, upload=475503]: Done uploading piece. err: <nil>
Dec 28 05:42:55.671830052 (542h13m55.542913087s) [noise=true , address=145.79.227.17:4970 , piece_id=CSPT4WC7HLSBBFMOMWKU23XPHUFQCGLFLSFMZTQWMNUA4D4QNY4Q, seg_pos={0 0}, upload=475503]: Done uploading piece. err: <nil>
Dec 28 05:42:55.681445682 (542h13m55.552528727s) [noise=true , address=145.79.227.15:4876 , piece_id=BMRS3CONOVJ6YTBJR735AALCRPTFV7G7S46MHWZ2RKX2KH3ZHHXA, seg_pos={0 0}, upload=475503]: Done uploading piece. err: <nil>
Dec 28 05:42:55.682097914 (542h13m55.553180959s) [noise=true , address=145.79.227.46:4603 , piece_id=BSK5JGLNRAOAEJ6FPYDE46JN5GSF7THRRUAY7YUAUI5NYCQTXLIQ, seg_pos={0 0}, upload=475503]: Done uploading piece. err: <nil>
Dec 28 05:42:55.688525404 (542h13m55.559608439s) [noise=true , address=145.79.227.11:4731 , piece_id=R3U3KCW5AXIUHODBCWZQMMERIM3NRKEQ4CFOBGLVFSS3QL3K2DLA, seg_pos={0 0}, upload=475503]: Done uploading piece. err: <nil>
Dec 28 05:42:55.700995659 (542h13m55.572078704s) [noise=true , address=145.79.227.36:4518 , piece_id=NRKTGUWXYI3D6SQ4PJRTY6FATLBFUU7LRFXYDJJMB5FLFDYVE32A, seg_pos={0 0}, upload=475503]: Done uploading piece. err: <nil>
Dec 28 05:42:55.922660725 (542h13m55.79374376s) [noise=true , address=45.91.116.14:4191 , piece_id=44LJ44JTITXMGACUSBV56ND5DGWA3GD4WNQB35K2OSZXOSYRPUJQ, seg_pos={0 0}, upload=475503]: Done uploading piece. err: <nil>
Dec 28 05:42:55.935179293 (542h13m55.806262338s) [noise=true , address=45.91.116.5:4299 , piece_id=BAVGDZER3LGWNTG47FDWJEA4MO57T6D3STMNPML75KXSNJOEJYEA, seg_pos={0 0}, upload=475503]: Done uploading piece. err: <nil>
Dec 28 05:42:55.935204881 (542h13m55.806287926s) [seg_pos={0 0}, upload=475503]: Segment reached optimal threshold of 13 pieces.
Dec 28 05:42:55.935486387 (542h13m55.806569432s) [noise=true , address=45.91.116.12:4225 , piece_id=TTULQP7LJP2XPJHG4GK7SM7TA3RP7IACVVWQXQGSFK2W2RIL5TNQ, seg_pos={0 0}, upload=475503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: read tcp 188.166.225.170:52920->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 28 05:42:55.935494222 (542h13m55.806577257s) [noise=true , address=45.91.116.12:4225 , piece_id=TTULQP7LJP2XPJHG4GK7SM7TA3RP7IACVVWQXQGSFK2W2RIL5TNQ, seg_pos={0 0}, upload=475503]: Long tail cancellation due to optimal threshold
Dec 28 05:42:55.935770517 (542h13m55.806853563s) [noise=true , address=45.91.116.9:4262 , piece_id=NCG3ZPSRM4HGNN4L4WCG5V7MOGERCM7H37NAR6ECJVIJNJIUIBNA, seg_pos={0 0}, upload=475503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF
Dec 28 05:42:55.935777572 (542h13m55.806860617s) [noise=true , address=45.91.116.9:4262 , piece_id=NCG3ZPSRM4HGNN4L4WCG5V7MOGERCM7H37NAR6ECJVIJNJIUIBNA, seg_pos={0 0}, upload=475503]: Long tail cancellation due to optimal threshold
Dec 28 05:42:55.935903831 (542h13m55.806986877s) [noise=true , address=45.91.116.6:4975 , piece_id=HJCXHJNNXEUXGUCTYOJ7OMQDNDLSJYDHQP4KS2KHSW4KF3YBU3QQ, seg_pos={0 0}, upload=475503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF
Dec 28 05:42:55.935928589 (542h13m55.807011644s) [noise=true , address=45.91.116.6:4975 , piece_id=HJCXHJNNXEUXGUCTYOJ7OMQDNDLSJYDHQP4KS2KHSW4KF3YBU3QQ, seg_pos={0 0}, upload=475503]: Long tail cancellation due to optimal threshold
Dec 28 05:42:55.935961442 (542h13m55.807044487s) [seg_pos={0 0}, upload=475503]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 05:42:55.936078745 (542h13m55.80716179s) [upload=475503]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 05:42:55.936085869 (542h13m55.807168914s) [upload=475503]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 05:42:56.241162713 (542h13m56.112245758s) [upload=475503]: Done uploading segments...
Dec 28 05:42:56.241179906 (542h13m56.112262941s) [upload=475503]: Upload finished. err: <nil>
Duration: 5.985008466s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
3 out of 60 is failed (threshold=3) [.........................X.........X...X....................] duration:
35.387076597s trace:
https://jaeger.dev.storj.io/trace/13fa7ae81fb54395
FAIL (2025-12-28 05:48:50 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
2LTTKX6TA, seg_pos={0 0}, upload=475338]: Uploading piece...
Dec 28 05:31:50.981939998 (542h2m50.853023033s) [noise=true , address=45.91.116.7:4038 , piece_id=XKTMNLAKBMYDBDGPF2OFELFZGZLKZKU6RH5HMPV5CK2XFGLR355A, seg_pos={0 0}, upload=475338]: Uploading piece...
Dec 28 05:31:50.982387089 (542h2m50.853470125s) [noise=true , address=45.91.116.4:4969 , piece_id=DP3NE6OD465KCCLBUSPM5CLDDCAHGBU5SMYQNSGBT6TPLJGWHQZQ, seg_pos={0 0}, upload=475338]: Uploading piece...
Dec 28 05:31:50.982882483 (542h2m50.853965528s) [noise=true , address=145.79.227.30:4790 , piece_id=NFUBU6YXRQNZPVYF2MQ5Z6SPNSDZKLDY6VTA3EZAGWBM2ISA4TFA, seg_pos={0 0}, upload=475338]: Uploading piece...
Dec 28 05:31:50.983279658 (542h2m50.854362704s) [noise=true , address=64.49.13.26:4949 , piece_id=K7HURP4RJP6C4EF32BFYNY3SW4SD7FCP5RTROCEDQ5DRTRFCKKQA, seg_pos={0 0}, upload=475338]: Uploading piece...
Dec 28 05:31:50.983672537 (542h2m50.854755582s) [noise=true , address=77.90.31.19:4859 , piece_id=IBMNSOOPT3V3KJ64BJR2HQIBX42PC5JJULCOLAAHRM5LFSE5T3KQ, seg_pos={0 0}, upload=475338]: Uploading piece...
Dec 28 05:31:50.984085172 (542h2m50.855168217s) [noise=true , address=77.90.31.17:4800 , piece_id=GCYS5VL5DPKRO72KZ42SU3UCJ7QWRKKVLZTM3PZJ5VQ73FT66XNQ, seg_pos={0 0}, upload=475338]: Uploading piece...
Dec 28 05:31:50.984529358 (542h2m50.855612403s) [noise=true , address=145.79.227.6:4267 , piece_id=5XIKAPK4XEH4JIRP54DG2RLHETKSIE2YGFB6CR6JHCCO5V4MRISQ, seg_pos={0 0}, upload=475338]: Uploading piece...
Dec 28 05:31:50.984943517 (542h2m50.856026562s) [noise=true , address=64.49.13.169:4708 , piece_id=UFVI7GQ7QMZJEZJ5W6GHUAYETSRBMQDBDEF2S5FWWDICFBMAUIQQ, seg_pos={0 0}, upload=475338]: Uploading piece...
Dec 28 05:31:50.990529894 (542h2m50.861612939s) [noise=true , address=145.79.227.25:4623 , piece_id=ZWQDLFTDPTSD6L3YNQRYCT3R2IXIYH7QV2MWMOHJDR3QHT7DHQLQ, seg_pos={0 0}, upload=475338]: Uploading piece...
Dec 28 05:31:50.992355541 (542h2m50.863438586s) [noise=true , address=145.79.227.43:4166 , piece_id=Y7PP5TLLB2AWB4TXOW2ZYDZYEWCN4V5UTNAD4OXUMOLLGOF44N7A, seg_pos={0 0}, upload=475338]: Uploading piece...
Dec 28 05:31:50.994108488 (542h2m50.865191533s) [noise=true , address=145.79.227.11:4744 , piece_id=V7GTO3MKQORI4TYRGQNH3HQU4MR3MM56LTARVUDQKGQDJ3DAXTZA, seg_pos={0 0}, upload=475338]: Uploading piece...
Dec 28 05:31:50.995802093 (542h2m50.866885138s) [noise=true , address=145.79.227.44:4197 , piece_id=XDW7YR65WGRXPXBH2IZHSF5YHAORSAHDLPIFIKYDDOI7MHDFYDVQ, seg_pos={0 0}, upload=475338]: Uploading piece...
Dec 28 05:31:50.997508242 (542h2m50.868591287s) [noise=true , address=77.90.31.21:4882 , piece_id=XEAHV4JSVJUU7HEQ3MWSNJDWXKY7SIAMQSXS3WMGZBSMULEOGV2A, seg_pos={0 0}, upload=475338]: Uploading piece...
Dec 28 05:31:50.999306125 (542h2m50.87038916s) [noise=true , address=145.79.227.33:4901 , piece_id=AFBX25C7QWLCQ5T3EJOGNOPDSJLKJHKMNCMMHCYSRYG3SCARROQQ, seg_pos={0 0}, upload=475338]: Uploading piece...
Dec 28 05:31:51.000524104 (542h2m50.87160716s) [noise=true , address=45.91.116.11:4136 , piece_id=OQAMHFOE3JDDW5BMWB3SMUUJAUBZRYQVKORYTCON6GO37FZQ5T3Q, seg_pos={0 0}, upload=475338]: Uploading piece...
Dec 28 05:31:52.349653213 (542h2m52.220736258s) [upload=475338]: Next returned nil segment
Dec 28 05:31:52.349668983 (542h2m52.220752018s) [upload=475338]: Waiting for error group managing segments...
Dec 28 05:31:52.698698340 (542h2m52.569781385s) [noise=true , address=64.49.13.26:4949 , piece_id=K7HURP4RJP6C4EF32BFYNY3SW4SD7FCP5RTROCEDQ5DRTRFCKKQA, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil>
Dec 28 05:31:53.070719678 (542h2m52.941802723s) [noise=true , address=77.90.31.15:4709 , piece_id=GWDIKXBLHVEL35QHHV36U5UV6CQYVP5FO7MKBFCKTQE2LTTKX6TA, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil>
Dec 28 05:31:53.082195551 (542h2m52.953278586s) [noise=true , address=45.91.116.11:4136 , piece_id=OQAMHFOE3JDDW5BMWB3SMUUJAUBZRYQVKORYTCON6GO37FZQ5T3Q, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil>
Dec 28 05:31:53.110615717 (542h2m52.981698772s) [noise=true , address=77.90.31.17:4800 , piece_id=GCYS5VL5DPKRO72KZ42SU3UCJ7QWRKKVLZTM3PZJ5VQ73FT66XNQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil>
Dec 28 05:31:53.192267075 (542h2m53.06335012s) [noise=true , address=45.91.116.7:4038 , piece_id=XKTMNLAKBMYDBDGPF2OFELFZGZLKZKU6RH5HMPV5CK2XFGLR355A, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil>
Dec 28 05:31:53.250763432 (542h2m53.121846477s) [noise=true , address=45.91.116.4:4969 , piece_id=DP3NE6OD465KCCLBUSPM5CLDDCAHGBU5SMYQNSGBT6TPLJGWHQZQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil>
Dec 28 05:31:53.764226697 (542h2m53.635309732s) [noise=true , address=77.90.31.19:4859 , piece_id=IBMNSOOPT3V3KJ64BJR2HQIBX42PC5JJULCOLAAHRM5LFSE5T3KQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil>
Dec 28 05:31:54.664687948 (542h2m54.535770984s) [noise=true , address=77.90.31.21:4882 , piece_id=XEAHV4JSVJUU7HEQ3MWSNJDWXKY7SIAMQSXS3WMGZBSMULEOGV2A, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil>
Dec 28 05:31:54.721529094 (542h2m54.592612139s) [noise=true , address=145.79.227.40:4072 , piece_id=NZRDWUF3LFOCZQ6YJ5TZHKU323Z2OCCKSDUYO2CLHZAT3NQTUEVA, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil>
Dec 28 05:31:54.910778063 (542h2m54.781861108s) [noise=true , address=145.79.227.30:4790 , piece_id=NFUBU6YXRQNZPVYF2MQ5Z6SPNSDZKLDY6VTA3EZAGWBM2ISA4TFA, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil>
Dec 28 05:31:55.419118326 (542h2m55.290201371s) [noise=true , address=145.79.227.44:4197 , piece_id=XDW7YR65WGRXPXBH2IZHSF5YHAORSAHDLPIFIKYDDOI7MHDFYDVQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil>
Dec 28 05:31:55.734980437 (542h2m55.606063482s) [noise=true , address=145.79.227.6:4267 , piece_id=5XIKAPK4XEH4JIRP54DG2RLHETKSIE2YGFB6CR6JHCCO5V4MRISQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil>
Dec 28 05:32:03.452240290 (542h3m3.323323336s) [noise=true , address=145.79.227.33:4901 , piece_id=AFBX25C7QWLCQ5T3EJOGNOPDSJLKJHKMNCMMHCYSRYG3SCARROQQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: ecclient: upload failed (node:1FHE84zvBb1MKwB1Ht7puTafgQxDEvPt4mYqkYVSSufAMpWQA1, address:145.79.227.33:4901): protocol: expected piece hash; hashstore: open /app/config/storage/hashstore/12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S/s0/cc/log-000000000006cecc-00000000: read-only file system; EOF
Dec 28 05:32:06.197217735 (542h3m6.06830079s) [noise=true , address=64.49.13.169:4708 , piece_id=UFVI7GQ7QMZJEZJ5W6GHUAYETSRBMQDBDEF2S5FWWDICFBMAUIQQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil>
Dec 28 05:32:48.823304697 (542h3m48.694387742s) [noise=true , address=145.79.227.43:4166 , piece_id=Y7PP5TLLB2AWB4TXOW2ZYDZYEWCN4V5UTNAD4OXUMOLLGOF44N7A, seg_pos={0 0}, upload=475338]: Done uploading piece. err: <nil>
Dec 28 05:32:48.823331638 (542h3m48.694414683s) [seg_pos={0 0}, upload=475338]: Segment reached optimal threshold of 14 pieces.
Dec 28 05:32:48.823588848 (542h3m48.694671883s) [noise=true , address=145.79.227.11:4744 , piece_id=V7GTO3MKQORI4TYRGQNH3HQU4MR3MM56LTARVUDQKGQDJ3DAXTZA, seg_pos={0 0}, upload=475338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A6abQMwnX4PhLYTvkGLinPx4eNeVyS15SGzWiLUq4cwKQ6bmH): protocol: write tcp 188.166.225.170:53066->145.79.227.11:4744: use of closed network connection; write tcp 188.166.225.170:53066->145.79.227.11:4744: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:53066->145.79.227.11:4744: use of closed network connection
Dec 28 05:32:48.823598617 (542h3m48.694681652s) [noise=true , address=145.79.227.11:4744 , piece_id=V7GTO3MKQORI4TYRGQNH3HQU4MR3MM56LTARVUDQKGQDJ3DAXTZA, seg_pos={0 0}, upload=475338]: Long tail cancellation due to optimal threshold
Dec 28 05:32:48.823769311 (542h3m48.694852356s) [noise=true , address=145.79.227.25:4623 , piece_id=ZWQDLFTDPTSD6L3YNQRYCT3R2IXIYH7QV2MWMOHJDR3QHT7DHQLQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eJPDT9UUjY1nszjYAjafGLFRrif8NjW3pxycFXPLRL2ZA2UG5): protocol: write tcp 188.166.225.170:60696->145.79.227.25:4623: use of closed network connection; write tcp 188.166.225.170:60696->145.79.227.25:4623: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60696->145.79.227.25:4623: use of closed network connection
Dec 28 05:32:48.823793027 (542h3m48.694876082s) [noise=true , address=145.79.227.25:4623 , piece_id=ZWQDLFTDPTSD6L3YNQRYCT3R2IXIYH7QV2MWMOHJDR3QHT7DHQLQ, seg_pos={0 0}, upload=475338]: Long tail cancellation due to optimal threshold
Dec 28 05:32:48.823914679 (542h3m48.694997724s) [noise=true , address=145.79.227.34:4940 , piece_id=TAGB23GFR2HT7KQBTNNKQMMYJKCDYTD2RELNTJNTUMDRUJWCSJKQ, seg_pos={0 0}, upload=475338]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ajfrKxYoBvXFurqvTkLDkiY39pAWGgmsMXfdBMVgDKvuhWRdr): protocol: expected piece hash; context canceled; EOF
Dec 28 05:32:48.823926431 (542h3m48.695009476s) [noise=true , address=145.79.227.34:4940 , piece_id=TAGB23GFR2HT7KQBTNNKQMMYJKCDYTD2RELNTJNTUMDRUJWCSJKQ, seg_pos={0 0}, upload=475338]: Long tail cancellation due to optimal threshold
Dec 28 05:32:48.823949335 (542h3m48.69503237s) [seg_pos={0 0}, upload=475338]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 28 05:32:48.824041500 (542h3m48.695124535s) [upload=475338]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 05:32:48.824047711 (542h3m48.695130756s) [upload=475338]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 05:32:49.147750383 (542h3m49.018833418s) [upload=475338]: Done uploading segments...
Dec 28 05:32:49.147769780 (542h3m49.018852816s) [upload=475338]: Upload finished. err: <nil>
Duration: 35.387096514s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
21 out of 60 is failed (threshold=3) [XX...X...X.XX.X.XX..XX.X...X.X.XX....X...X.X..X.....X.......] duration:
5.655526189s trace:
https://jaeger.dev.storj.io/trace/58bbbbe9bef3e1af
FAIL (2025-12-28 05:48:10 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 28 05:42:53.387982259 (542h13m53.259065304s) [upload=475503]: Starting upload
Dec 28 05:42:53.388039338 (542h13m53.259122384s) [upload=475503]: Done starting upload
Dec 28 05:42:53.388153736 (542h13m53.259236781s) [upload=475503]: Uploading segments...
Dec 28 05:42:53.388318510 (542h13m53.259401555s) [upload=475503]: Got next segment. Inline: false
Dec 28 05:42:53.388357434 (542h13m53.259440469s) [upload=475503]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 28 05:42:53.388363225 (542h13m53.25944627s) [upload=475503]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 28 05:42:53.640155558 (542h13m53.511238603s) [seg_pos={0 0}, upload=475503]: Begin upload segment...
Dec 28 05:42:53.640310503 (542h13m53.511393548s) [seg_pos={0 0}, upload=475503]: Done begin upload segment.
Dec 28 05:42:53.640394433 (542h13m53.511477468s) [noise=true , address=145.79.227.2:4115 , piece_id=AALV2IIJHUEG4GTR5OQFTRN2F4DT5WYUOXELRTUU3NWC6ETFLFWQ, seg_pos={0 0}, upload=475503]: Uploading piece...
Dec 28 05:42:53.640435691 (542h13m53.511518726s) [noise=true , address=45.91.116.9:4262 , piece_id=NCG3ZPSRM4HGNN4L4WCG5V7MOGERCM7H37NAR6ECJVIJNJIUIBNA, seg_pos={0 0}, upload=475503]: Uploading piece...
Dec 28 05:42:53.641347858 (542h13m53.512430893s) [noise=true , address=145.79.227.37:4567 , piece_id=E2KSUSOQMK37YVP3G6FKFJ64RJXRMP7Q72HTEUVVJOVUSB33TRQA, seg_pos={0 0}, upload=475503]: Uploading piece...
Dec 28 05:42:53.642303769 (542h13m53.513386804s) [noise=true , address=145.79.227.31:4798 , piece_id=NXUROFV5C47O2LJFARP5FVO2B765MZKE6UNUMSEJIMBESKBQT6MA, seg_pos={0 0}, upload=475503]: Uploading piece...
Dec 28 05:42:53.643155851 (542h13m53.514238896s) [noise=true , address=145.79.227.15:4876 , piece_id=BMRS3CONOVJ6YTBJR735AALCRPTFV7G7S46MHWZ2RKX2KH3ZHHXA, seg_pos={0 0}, upload=475503]: Uploading piece...
Dec 28 05:42:53.644484592 (542h13m53.515567627s) [noise=true , address=145.79.227.36:4518 , piece_id=NRKTGUWXYI3D6SQ4PJRTY6FATLBFUU7LRFXYDJJMB5FLFDYVE32A, seg_pos={0 0}, upload=475503]: Uploading piece...
Dec 28 05:42:53.646190490 (542h13m53.517273535s) [noise=true , address=45.91.116.6:4975 , piece_id=HJCXHJNNXEUXGUCTYOJ7OMQDNDLSJYDHQP4KS2KHSW4KF3YBU3QQ, seg_pos={0 0}, upload=475503]: Uploading piece...
Dec 28 05:42:53.648465171 (542h13m53.519548216s) [noise=true , address=145.79.227.52:4445 , piece_id=A5TCQ2GV5Q7P6SUF6PK4BHF7SAYGOTAFENV5GAQ4I6NDFNWD2OBQ, seg_pos={0 0}, upload=475503]: Uploading piece...
Dec 28 05:42:53.651026908 (542h13m53.522109954s) [noise=true , address=145.79.227.11:4731 , piece_id=R3U3KCW5AXIUHODBCWZQMMERIM3NRKEQ4CFOBGLVFSS3QL3K2DLA, seg_pos={0 0}, upload=475503]: Uploading piece...
Dec 28 05:42:53.651705571 (542h13m53.522788606s) [noise=true , address=145.79.227.46:4603 , piece_id=BSK5JGLNRAOAEJ6FPYDE46JN5GSF7THRRUAY7YUAUI5NYCQTXLIQ, seg_pos={0 0}, upload=475503]: Uploading piece...
Dec 28 05:42:53.653001268 (542h13m53.524084303s) [noise=true , address=45.91.116.14:4191 , piece_id=44LJ44JTITXMGACUSBV56ND5DGWA3GD4WNQB35K2OSZXOSYRPUJQ, seg_pos={0 0}, upload=475503]: Uploading piece...
Dec 28 05:42:53.654305932 (542h13m53.525388977s) [noise=true , address=145.79.227.50:4374 , piece_id=5DIZD6MCNYMVTBWP6CPB74NVSQB72J6MVTW36EG6QQEO5J4OO3MA, seg_pos={0 0}, upload=475503]: Uploading piece...
Dec 28 05:42:53.656135045 (542h13m53.527218091s) [noise=true , address=145.79.227.32:4835 , piece_id=37M4346GVUKBGONLZ662ITUZIA65AIFEXL5KGONXX3EEP4JSWO3A, seg_pos={0 0}, upload=475503]: Uploading piece...
Dec 28 05:42:53.657413991 (542h13m53.528497036s) [noise=true , address=45.91.116.5:4299 , piece_id=BAVGDZER3LGWNTG47FDWJEA4MO57T6D3STMNPML75KXSNJOEJYEA, seg_pos={0 0}, upload=475503]: Uploading piece...
Dec 28 05:42:53.657754859 (542h13m53.528837904s) [noise=true , address=45.91.116.12:4225 , piece_id=TTULQP7LJP2XPJHG4GK7SM7TA3RP7IACVVWQXQGSFK2W2RIL5TNQ, seg_pos={0 0}, upload=475503]: Uploading piece...
Dec 28 05:42:53.659438286 (542h13m53.530521331s) [noise=true , address=145.79.227.17:4970 , piece_id=CSPT4WC7HLSBBFMOMWKU23XPHUFQCGLFLSFMZTQWMNUA4D4QNY4Q, seg_pos={0 0}, upload=475503]: Uploading piece...
Dec 28 05:42:55.360476706 (542h13m55.231559751s) [upload=475503]: Next returned nil segment
Dec 28 05:42:55.360555025 (542h13m55.23163807s) [upload=475503]: Waiting for error group managing segments...
Dec 28 05:42:55.652562073 (542h13m55.523645118s) [noise=true , address=145.79.227.32:4835 , piece_id=37M4346GVUKBGONLZ662ITUZIA65AIFEXL5KGONXX3EEP4JSWO3A, seg_pos={0 0}, upload=475503]: Done uploading piece. err: <nil>
Dec 28 05:42:55.652599064 (542h13m55.523682129s) [noise=true , address=145.79.227.2:4115 , piece_id=AALV2IIJHUEG4GTR5OQFTRN2F4DT5WYUOXELRTUU3NWC6ETFLFWQ, seg_pos={0 0}, upload=475503]: Done uploading piece. err: <nil>
Dec 28 05:42:55.657515976 (542h13m55.528599021s) [noise=true , address=145.79.227.52:4445 , piece_id=A5TCQ2GV5Q7P6SUF6PK4BHF7SAYGOTAFENV5GAQ4I6NDFNWD2OBQ, seg_pos={0 0}, upload=475503]: Done uploading piece. err: <nil>
Dec 28 05:42:55.657674639 (542h13m55.528757684s) [noise=true , address=145.79.227.31:4798 , piece_id=NXUROFV5C47O2LJFARP5FVO2B765MZKE6UNUMSEJIMBESKBQT6MA, seg_pos={0 0}, upload=475503]: Done uploading piece. err: <nil>
Dec 28 05:42:55.658848644 (542h13m55.52993168s) [noise=true , address=145.79.227.37:4567 , piece_id=E2KSUSOQMK37YVP3G6FKFJ64RJXRMP7Q72HTEUVVJOVUSB33TRQA, seg_pos={0 0}, upload=475503]: Done uploading piece. err: <nil>
Dec 28 05:42:55.663002935 (542h13m55.53408597s) [noise=true , address=145.79.227.50:4374 , piece_id=5DIZD6MCNYMVTBWP6CPB74NVSQB72J6MVTW36EG6QQEO5J4OO3MA, seg_pos={0 0}, upload=475503]: Done uploading piece. err: <nil>
Dec 28 05:42:55.671830052 (542h13m55.542913087s) [noise=true , address=145.79.227.17:4970 , piece_id=CSPT4WC7HLSBBFMOMWKU23XPHUFQCGLFLSFMZTQWMNUA4D4QNY4Q, seg_pos={0 0}, upload=475503]: Done uploading piece. err: <nil>
Dec 28 05:42:55.681445682 (542h13m55.552528727s) [noise=true , address=145.79.227.15:4876 , piece_id=BMRS3CONOVJ6YTBJR735AALCRPTFV7G7S46MHWZ2RKX2KH3ZHHXA, seg_pos={0 0}, upload=475503]: Done uploading piece. err: <nil>
Dec 28 05:42:55.682097914 (542h13m55.553180959s) [noise=true , address=145.79.227.46:4603 , piece_id=BSK5JGLNRAOAEJ6FPYDE46JN5GSF7THRRUAY7YUAUI5NYCQTXLIQ, seg_pos={0 0}, upload=475503]: Done uploading piece. err: <nil>
Dec 28 05:42:55.688525404 (542h13m55.559608439s) [noise=true , address=145.79.227.11:4731 , piece_id=R3U3KCW5AXIUHODBCWZQMMERIM3NRKEQ4CFOBGLVFSS3QL3K2DLA, seg_pos={0 0}, upload=475503]: Done uploading piece. err: <nil>
Dec 28 05:42:55.700995659 (542h13m55.572078704s) [noise=true , address=145.79.227.36:4518 , piece_id=NRKTGUWXYI3D6SQ4PJRTY6FATLBFUU7LRFXYDJJMB5FLFDYVE32A, seg_pos={0 0}, upload=475503]: Done uploading piece. err: <nil>
Dec 28 05:42:55.922660725 (542h13m55.79374376s) [noise=true , address=45.91.116.14:4191 , piece_id=44LJ44JTITXMGACUSBV56ND5DGWA3GD4WNQB35K2OSZXOSYRPUJQ, seg_pos={0 0}, upload=475503]: Done uploading piece. err: <nil>
Dec 28 05:42:55.935179293 (542h13m55.806262338s) [noise=true , address=45.91.116.5:4299 , piece_id=BAVGDZER3LGWNTG47FDWJEA4MO57T6D3STMNPML75KXSNJOEJYEA, seg_pos={0 0}, upload=475503]: Done uploading piece. err: <nil>
Dec 28 05:42:55.935204881 (542h13m55.806287926s) [seg_pos={0 0}, upload=475503]: Segment reached optimal threshold of 13 pieces.
Dec 28 05:42:55.935486387 (542h13m55.806569432s) [noise=true , address=45.91.116.12:4225 , piece_id=TTULQP7LJP2XPJHG4GK7SM7TA3RP7IACVVWQXQGSFK2W2RIL5TNQ, seg_pos={0 0}, upload=475503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: read tcp 188.166.225.170:52920->45.91.116.12:4225: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 28 05:42:55.935494222 (542h13m55.806577257s) [noise=true , address=45.91.116.12:4225 , piece_id=TTULQP7LJP2XPJHG4GK7SM7TA3RP7IACVVWQXQGSFK2W2RIL5TNQ, seg_pos={0 0}, upload=475503]: Long tail cancellation due to optimal threshold
Dec 28 05:42:55.935770517 (542h13m55.806853563s) [noise=true , address=45.91.116.9:4262 , piece_id=NCG3ZPSRM4HGNN4L4WCG5V7MOGERCM7H37NAR6ECJVIJNJIUIBNA, seg_pos={0 0}, upload=475503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Rebnup7LaVo85uFG24Afr6oReVAc5NiAARKApQaP8soPczPto): protocol: expected piece hash; context canceled; EOF
Dec 28 05:42:55.935777572 (542h13m55.806860617s) [noise=true , address=45.91.116.9:4262 , piece_id=NCG3ZPSRM4HGNN4L4WCG5V7MOGERCM7H37NAR6ECJVIJNJIUIBNA, seg_pos={0 0}, upload=475503]: Long tail cancellation due to optimal threshold
Dec 28 05:42:55.935903831 (542h13m55.806986877s) [noise=true , address=45.91.116.6:4975 , piece_id=HJCXHJNNXEUXGUCTYOJ7OMQDNDLSJYDHQP4KS2KHSW4KF3YBU3QQ, seg_pos={0 0}, upload=475503]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7): protocol: expected piece hash; context canceled; EOF
Dec 28 05:42:55.935928589 (542h13m55.807011644s) [noise=true , address=45.91.116.6:4975 , piece_id=HJCXHJNNXEUXGUCTYOJ7OMQDNDLSJYDHQP4KS2KHSW4KF3YBU3QQ, seg_pos={0 0}, upload=475503]: Long tail cancellation due to optimal threshold
Dec 28 05:42:55.935961442 (542h13m55.807044487s) [seg_pos={0 0}, upload=475503]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 28 05:42:55.936078745 (542h13m55.80716179s) [upload=475503]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 28 05:42:55.936085869 (542h13m55.807168914s) [upload=475503]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 28 05:42:56.241162713 (542h13m56.112245758s) [upload=475503]: Done uploading segments...
Dec 28 05:42:56.241179906 (542h13m56.112262941s) [upload=475503]: Upload finished. err: <nil>
Duration: 5.655547859s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
26 out of 60 is failed (threshold=6) [X.X.XX.XXX.....XXX....XXXX....XXX.X...X....X...XX.X..XXX....] duration:
4.750866912s trace:
https://jaeger.dev.storj.io/trace/5cb5db0cf501010b
FAIL (2025-12-28 05:48:09 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.750893493s
core
uplink
us-select-1
us1-placement-20
|