Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X.........X.............................................X..] duration: 5.9378177s trace: https://jaeger.dev.storj.io/trace/1ed3345d7ee41336 FAIL (2025-12-23 17:10:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 23 17:09:26.380966455 (433h40m26.2520495s) [upload=380680]: Starting upload Dec 23 17:09:26.381043451 (433h40m26.252126496s) [upload=380680]: Done starting upload Dec 23 17:09:26.381132912 (433h40m26.252215957s) [upload=380680]: Uploading segments... Dec 23 17:09:26.381368230 (433h40m26.252451275s) [upload=380680]: Got next segment. Inline: false Dec 23 17:09:26.381415329 (433h40m26.252498374s) [upload=380680]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 17:09:26.381421922 (433h40m26.252504977s) [upload=380680]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 17:09:26.628163610 (433h40m26.499246666s) [seg_pos={0 0}, upload=380680]: Begin upload segment... Dec 23 17:09:26.628275714 (433h40m26.499358749s) [seg_pos={0 0}, upload=380680]: Done begin upload segment. Dec 23 17:09:26.628350947 (433h40m26.499433992s) [noise=true , address=145.79.227.25:4617 , piece_id=DIQKCB3XRWK7RFEUZLGEHM3JX5LSD7PM2BHF2WSCMT3RH2P57HBQ, seg_pos={0 0}, upload=380680]: Uploading piece... Dec 23 17:09:26.628370284 (433h40m26.499453329s) [noise=true , address=145.79.227.52:4446 , piece_id=KT7GN3Q3THGLBREEOCC4MOG5DLHFD2LLGAOMNNXBOGPWUFHMSWFA, seg_pos={0 0}, upload=380680]: Uploading piece... Dec 23 17:09:26.628914840 (433h40m26.499997885s) [noise=true , address=145.79.227.7:4294 , piece_id=KIA4I2FJXUQELXOIQMNEZIOOBJTKZHUWX2H6YYJNCIFJWFDEGLCQ, seg_pos={0 0}, upload=380680]: Uploading piece... Dec 23 17:09:26.628931663 (433h40m26.500014698s) [noise=true , address=145.79.227.13:4804 , piece_id=BRMAM2YZMECAQRC7IRO57MW7HV6QY2MQ3WV6C6MN5NEYRAZV7JHA, seg_pos={0 0}, upload=380680]: Uploading piece... Dec 23 17:09:26.629249067 (433h40m26.500332102s) [noise=true , address=45.91.116.5:4298 , piece_id=AZD5RY7IMH4IIQ5CNBX3WZXVUD4SOVZ34QMHTYFN63N23OQNBITQ, seg_pos={0 0}, upload=380680]: Uploading piece... Dec 23 17:09:26.629340752 (433h40m26.500423797s) [noise=true , address=77.90.31.11:4555 , piece_id=L6TXVDKRR22ARSKVOXEU7T56YXPIUSEU5S23K23X3SP6XBGIY2LA, seg_pos={0 0}, upload=380680]: Uploading piece... Dec 23 17:09:26.629776722 (433h40m26.500859767s) [noise=true , address=145.79.227.37:4565 , piece_id=TJTD73TKDSXYUATA5JPKUZDBKXPCWKAIGW75QJU2W5MWYCGJYMUA, seg_pos={0 0}, upload=380680]: Uploading piece... Dec 23 17:09:26.630315418 (433h40m26.501398453s) [noise=true , address=145.79.227.48:4303 , piece_id=WTOINPDQXQ44R7F23TSV3SU6SJW7ELGWU4QKSMV5I7NWNMFL4NJA, seg_pos={0 0}, upload=380680]: Uploading piece... Dec 23 17:09:26.630678309 (433h40m26.501761344s) [noise=true , address=145.79.227.46:4601 , piece_id=4NNF2YLY7ZCONKV4CPPQGVLRMFEGENPZO3ZL6MLZ7XYFWJ4ZCTEQ, seg_pos={0 0}, upload=380680]: Uploading piece... Dec 23 17:09:26.630847040 (433h40m26.501930085s) [noise=true , address=145.79.227.42:4121 , piece_id=WTZSX53JJMAENNS2FVBBNSBGOIRNTF7HMKDP4ADYVWKNROOA32KQ, seg_pos={0 0}, upload=380680]: Uploading piece... Dec 23 17:09:26.631137122 (433h40m26.502220157s) [noise=true , address=145.79.227.23:4545 , piece_id=AX2MU4GIIHT7VREYS3GVHMFNLYAYEZ7ZOGWTKHA37Q7UCGR5BKZQ, seg_pos={0 0}, upload=380680]: Uploading piece... Dec 23 17:09:26.631588452 (433h40m26.502671497s) [noise=true , address=45.91.116.8:4048 , piece_id=DENFYN3LEVL2PKTFM6TJBJKKDX5COUXRL5PPLACS7WVEHPJ5PYWQ, seg_pos={0 0}, upload=380680]: Uploading piece... Dec 23 17:09:26.631766671 (433h40m26.502849716s) [noise=true , address=145.79.227.33:4871 , piece_id=UOKKYR7PEZRFHG62ULTCA67I4C7VVDW44G7JRVLANORDQS5P5YYQ, seg_pos={0 0}, upload=380680]: Uploading piece... Dec 23 17:09:26.632168226 (433h40m26.503251271s) [noise=true , address=145.79.227.43:4159 , piece_id=O6VZTNE3YU6PXK3WO7JXNC3RY5ISEZ2H4COWBVKAJ4RDLWXAW3VA, seg_pos={0 0}, upload=380680]: Uploading piece... Dec 23 17:09:26.632602562 (433h40m26.503685597s) [noise=true , address=145.79.227.11:4729 , piece_id=QDHMDK6FZ5IZIQGKCTNB7SCAPGKYYA5IUPDZZXKPMTR4AXWP7YJA, seg_pos={0 0}, upload=380680]: Uploading piece... Dec 23 17:09:26.632932121 (433h40m26.504015166s) [noise=true , address=45.91.116.7:4009 , piece_id=7F4EFJ63IWKAB6DKBZYHQ6656T474EO2VZB45W2ILXQR7T3KTIUQ, seg_pos={0 0}, upload=380680]: Uploading piece... Dec 23 17:09:28.252996980 (433h40m28.124080016s) [upload=380680]: Next returned nil segment Dec 23 17:09:28.253035976 (433h40m28.124119021s) [upload=380680]: Waiting for error group managing segments... Dec 23 17:09:28.637647661 (433h40m28.508730706s) [noise=true , address=145.79.227.46:4601 , piece_id=4NNF2YLY7ZCONKV4CPPQGVLRMFEGENPZO3ZL6MLZ7XYFWJ4ZCTEQ, seg_pos={0 0}, upload=380680]: Done uploading piece. err: <nil> Dec 23 17:09:28.646150863 (433h40m28.517233908s) [noise=true , address=145.79.227.23:4545 , piece_id=AX2MU4GIIHT7VREYS3GVHMFNLYAYEZ7ZOGWTKHA37Q7UCGR5BKZQ, seg_pos={0 0}, upload=380680]: Done uploading piece. err: <nil> Dec 23 17:09:28.651389007 (433h40m28.522472052s) [noise=true , address=145.79.227.25:4617 , piece_id=DIQKCB3XRWK7RFEUZLGEHM3JX5LSD7PM2BHF2WSCMT3RH2P57HBQ, seg_pos={0 0}, upload=380680]: Done uploading piece. err: <nil> Dec 23 17:09:28.651650015 (433h40m28.52273307s) [noise=true , address=145.79.227.43:4159 , piece_id=O6VZTNE3YU6PXK3WO7JXNC3RY5ISEZ2H4COWBVKAJ4RDLWXAW3VA, seg_pos={0 0}, upload=380680]: Done uploading piece. err: <nil> Dec 23 17:09:28.652242443 (433h40m28.523325478s) [noise=true , address=145.79.227.7:4294 , piece_id=KIA4I2FJXUQELXOIQMNEZIOOBJTKZHUWX2H6YYJNCIFJWFDEGLCQ, seg_pos={0 0}, upload=380680]: Done uploading piece. err: <nil> Dec 23 17:09:28.652411425 (433h40m28.52349446s) [noise=true , address=145.79.227.11:4729 , piece_id=QDHMDK6FZ5IZIQGKCTNB7SCAPGKYYA5IUPDZZXKPMTR4AXWP7YJA, seg_pos={0 0}, upload=380680]: Done uploading piece. err: <nil> Dec 23 17:09:28.652943417 (433h40m28.524026462s) [noise=true , address=145.79.227.13:4804 , piece_id=BRMAM2YZMECAQRC7IRO57MW7HV6QY2MQ3WV6C6MN5NEYRAZV7JHA, seg_pos={0 0}, upload=380680]: Done uploading piece. err: <nil> Dec 23 17:09:28.653647879 (433h40m28.524730924s) [noise=true , address=145.79.227.37:4565 , piece_id=TJTD73TKDSXYUATA5JPKUZDBKXPCWKAIGW75QJU2W5MWYCGJYMUA, seg_pos={0 0}, upload=380680]: Done uploading piece. err: <nil> Dec 23 17:09:28.654077386 (433h40m28.525160421s) [noise=true , address=145.79.227.52:4446 , piece_id=KT7GN3Q3THGLBREEOCC4MOG5DLHFD2LLGAOMNNXBOGPWUFHMSWFA, seg_pos={0 0}, upload=380680]: Done uploading piece. err: <nil> Dec 23 17:09:28.655538408 (433h40m28.526621453s) [noise=true , address=145.79.227.42:4121 , piece_id=WTZSX53JJMAENNS2FVBBNSBGOIRNTF7HMKDP4ADYVWKNROOA32KQ, seg_pos={0 0}, upload=380680]: Done uploading piece. err: <nil> Dec 23 17:09:28.659683211 (433h40m28.530766257s) [noise=true , address=145.79.227.48:4303 , piece_id=WTOINPDQXQ44R7F23TSV3SU6SJW7ELGWU4QKSMV5I7NWNMFL4NJA, seg_pos={0 0}, upload=380680]: Done uploading piece. err: <nil> Dec 23 17:09:28.904599935 (433h40m28.77568297s) [noise=true , address=77.90.31.11:4555 , piece_id=L6TXVDKRR22ARSKVOXEU7T56YXPIUSEU5S23K23X3SP6XBGIY2LA, seg_pos={0 0}, upload=380680]: Done uploading piece. err: <nil> Dec 23 17:09:28.918645880 (433h40m28.789728925s) [noise=true , address=45.91.116.7:4009 , piece_id=7F4EFJ63IWKAB6DKBZYHQ6656T474EO2VZB45W2ILXQR7T3KTIUQ, seg_pos={0 0}, upload=380680]: Done uploading piece. err: <nil> Dec 23 17:09:28.918676428 (433h40m28.789759473s) [seg_pos={0 0}, upload=380680]: Segment reached optimal threshold of 13 pieces. Dec 23 17:09:28.918920483 (433h40m28.790003518s) [noise=true , address=45.91.116.8:4048 , piece_id=DENFYN3LEVL2PKTFM6TJBJKKDX5COUXRL5PPLACS7WVEHPJ5PYWQ, seg_pos={0 0}, upload=380680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 23 17:09:28.918928338 (433h40m28.790011373s) [noise=true , address=45.91.116.8:4048 , piece_id=DENFYN3LEVL2PKTFM6TJBJKKDX5COUXRL5PPLACS7WVEHPJ5PYWQ, seg_pos={0 0}, upload=380680]: Long tail cancellation due to optimal threshold Dec 23 17:09:28.919073044 (433h40m28.790156089s) [noise=true , address=45.91.116.5:4298 , piece_id=AZD5RY7IMH4IIQ5CNBX3WZXVUD4SOVZ34QMHTYFN63N23OQNBITQ, seg_pos={0 0}, upload=380680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 23 17:09:28.919079085 (433h40m28.79016212s) [noise=true , address=45.91.116.5:4298 , piece_id=AZD5RY7IMH4IIQ5CNBX3WZXVUD4SOVZ34QMHTYFN63N23OQNBITQ, seg_pos={0 0}, upload=380680]: Long tail cancellation due to optimal threshold Dec 23 17:09:28.919314434 (433h40m28.790397479s) [noise=true , address=145.79.227.33:4871 , piece_id=UOKKYR7PEZRFHG62ULTCA67I4C7VVDW44G7JRVLANORDQS5P5YYQ, seg_pos={0 0}, upload=380680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 23 17:09:28.919361253 (433h40m28.790444298s) [noise=true , address=145.79.227.33:4871 , piece_id=UOKKYR7PEZRFHG62ULTCA67I4C7VVDW44G7JRVLANORDQS5P5YYQ, seg_pos={0 0}, upload=380680]: Long tail cancellation due to optimal threshold Dec 23 17:09:28.919398023 (433h40m28.790481068s) [seg_pos={0 0}, upload=380680]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 17:09:28.919611008 (433h40m28.790694053s) [upload=380680]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 17:09:28.919621768 (433h40m28.790704813s) [upload=380680]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 17:09:29.220706044 (433h40m29.091789089s) [upload=380680]: Done uploading segments... Dec 23 17:09:29.220727275 (433h40m29.09181032s) [upload=380680]: Upload finished. err: <nil> Duration: 5.937835585s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X.........X.............................................X.] duration: 5.818237731s trace: https://jaeger.dev.storj.io/trace/1ed3345d7ee41336 FAIL (2025-12-23 17:10:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 23 17:09:26.380966455 (433h40m26.2520495s) [upload=380680]: Starting upload Dec 23 17:09:26.381043451 (433h40m26.252126496s) [upload=380680]: Done starting upload Dec 23 17:09:26.381132912 (433h40m26.252215957s) [upload=380680]: Uploading segments... Dec 23 17:09:26.381368230 (433h40m26.252451275s) [upload=380680]: Got next segment. Inline: false Dec 23 17:09:26.381415329 (433h40m26.252498374s) [upload=380680]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 17:09:26.381421922 (433h40m26.252504977s) [upload=380680]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 17:09:26.628163610 (433h40m26.499246666s) [seg_pos={0 0}, upload=380680]: Begin upload segment... Dec 23 17:09:26.628275714 (433h40m26.499358749s) [seg_pos={0 0}, upload=380680]: Done begin upload segment. Dec 23 17:09:26.628350947 (433h40m26.499433992s) [noise=true , address=145.79.227.25:4617 , piece_id=DIQKCB3XRWK7RFEUZLGEHM3JX5LSD7PM2BHF2WSCMT3RH2P57HBQ, seg_pos={0 0}, upload=380680]: Uploading piece... Dec 23 17:09:26.628370284 (433h40m26.499453329s) [noise=true , address=145.79.227.52:4446 , piece_id=KT7GN3Q3THGLBREEOCC4MOG5DLHFD2LLGAOMNNXBOGPWUFHMSWFA, seg_pos={0 0}, upload=380680]: Uploading piece... Dec 23 17:09:26.628914840 (433h40m26.499997885s) [noise=true , address=145.79.227.7:4294 , piece_id=KIA4I2FJXUQELXOIQMNEZIOOBJTKZHUWX2H6YYJNCIFJWFDEGLCQ, seg_pos={0 0}, upload=380680]: Uploading piece... Dec 23 17:09:26.628931663 (433h40m26.500014698s) [noise=true , address=145.79.227.13:4804 , piece_id=BRMAM2YZMECAQRC7IRO57MW7HV6QY2MQ3WV6C6MN5NEYRAZV7JHA, seg_pos={0 0}, upload=380680]: Uploading piece... Dec 23 17:09:26.629249067 (433h40m26.500332102s) [noise=true , address=45.91.116.5:4298 , piece_id=AZD5RY7IMH4IIQ5CNBX3WZXVUD4SOVZ34QMHTYFN63N23OQNBITQ, seg_pos={0 0}, upload=380680]: Uploading piece... Dec 23 17:09:26.629340752 (433h40m26.500423797s) [noise=true , address=77.90.31.11:4555 , piece_id=L6TXVDKRR22ARSKVOXEU7T56YXPIUSEU5S23K23X3SP6XBGIY2LA, seg_pos={0 0}, upload=380680]: Uploading piece... Dec 23 17:09:26.629776722 (433h40m26.500859767s) [noise=true , address=145.79.227.37:4565 , piece_id=TJTD73TKDSXYUATA5JPKUZDBKXPCWKAIGW75QJU2W5MWYCGJYMUA, seg_pos={0 0}, upload=380680]: Uploading piece... Dec 23 17:09:26.630315418 (433h40m26.501398453s) [noise=true , address=145.79.227.48:4303 , piece_id=WTOINPDQXQ44R7F23TSV3SU6SJW7ELGWU4QKSMV5I7NWNMFL4NJA, seg_pos={0 0}, upload=380680]: Uploading piece... Dec 23 17:09:26.630678309 (433h40m26.501761344s) [noise=true , address=145.79.227.46:4601 , piece_id=4NNF2YLY7ZCONKV4CPPQGVLRMFEGENPZO3ZL6MLZ7XYFWJ4ZCTEQ, seg_pos={0 0}, upload=380680]: Uploading piece... Dec 23 17:09:26.630847040 (433h40m26.501930085s) [noise=true , address=145.79.227.42:4121 , piece_id=WTZSX53JJMAENNS2FVBBNSBGOIRNTF7HMKDP4ADYVWKNROOA32KQ, seg_pos={0 0}, upload=380680]: Uploading piece... Dec 23 17:09:26.631137122 (433h40m26.502220157s) [noise=true , address=145.79.227.23:4545 , piece_id=AX2MU4GIIHT7VREYS3GVHMFNLYAYEZ7ZOGWTKHA37Q7UCGR5BKZQ, seg_pos={0 0}, upload=380680]: Uploading piece... Dec 23 17:09:26.631588452 (433h40m26.502671497s) [noise=true , address=45.91.116.8:4048 , piece_id=DENFYN3LEVL2PKTFM6TJBJKKDX5COUXRL5PPLACS7WVEHPJ5PYWQ, seg_pos={0 0}, upload=380680]: Uploading piece... Dec 23 17:09:26.631766671 (433h40m26.502849716s) [noise=true , address=145.79.227.33:4871 , piece_id=UOKKYR7PEZRFHG62ULTCA67I4C7VVDW44G7JRVLANORDQS5P5YYQ, seg_pos={0 0}, upload=380680]: Uploading piece... Dec 23 17:09:26.632168226 (433h40m26.503251271s) [noise=true , address=145.79.227.43:4159 , piece_id=O6VZTNE3YU6PXK3WO7JXNC3RY5ISEZ2H4COWBVKAJ4RDLWXAW3VA, seg_pos={0 0}, upload=380680]: Uploading piece... Dec 23 17:09:26.632602562 (433h40m26.503685597s) [noise=true , address=145.79.227.11:4729 , piece_id=QDHMDK6FZ5IZIQGKCTNB7SCAPGKYYA5IUPDZZXKPMTR4AXWP7YJA, seg_pos={0 0}, upload=380680]: Uploading piece... Dec 23 17:09:26.632932121 (433h40m26.504015166s) [noise=true , address=45.91.116.7:4009 , piece_id=7F4EFJ63IWKAB6DKBZYHQ6656T474EO2VZB45W2ILXQR7T3KTIUQ, seg_pos={0 0}, upload=380680]: Uploading piece... Dec 23 17:09:28.252996980 (433h40m28.124080016s) [upload=380680]: Next returned nil segment Dec 23 17:09:28.253035976 (433h40m28.124119021s) [upload=380680]: Waiting for error group managing segments... Dec 23 17:09:28.637647661 (433h40m28.508730706s) [noise=true , address=145.79.227.46:4601 , piece_id=4NNF2YLY7ZCONKV4CPPQGVLRMFEGENPZO3ZL6MLZ7XYFWJ4ZCTEQ, seg_pos={0 0}, upload=380680]: Done uploading piece. err: <nil> Dec 23 17:09:28.646150863 (433h40m28.517233908s) [noise=true , address=145.79.227.23:4545 , piece_id=AX2MU4GIIHT7VREYS3GVHMFNLYAYEZ7ZOGWTKHA37Q7UCGR5BKZQ, seg_pos={0 0}, upload=380680]: Done uploading piece. err: <nil> Dec 23 17:09:28.651389007 (433h40m28.522472052s) [noise=true , address=145.79.227.25:4617 , piece_id=DIQKCB3XRWK7RFEUZLGEHM3JX5LSD7PM2BHF2WSCMT3RH2P57HBQ, seg_pos={0 0}, upload=380680]: Done uploading piece. err: <nil> Dec 23 17:09:28.651650015 (433h40m28.52273307s) [noise=true , address=145.79.227.43:4159 , piece_id=O6VZTNE3YU6PXK3WO7JXNC3RY5ISEZ2H4COWBVKAJ4RDLWXAW3VA, seg_pos={0 0}, upload=380680]: Done uploading piece. err: <nil> Dec 23 17:09:28.652242443 (433h40m28.523325478s) [noise=true , address=145.79.227.7:4294 , piece_id=KIA4I2FJXUQELXOIQMNEZIOOBJTKZHUWX2H6YYJNCIFJWFDEGLCQ, seg_pos={0 0}, upload=380680]: Done uploading piece. err: <nil> Dec 23 17:09:28.652411425 (433h40m28.52349446s) [noise=true , address=145.79.227.11:4729 , piece_id=QDHMDK6FZ5IZIQGKCTNB7SCAPGKYYA5IUPDZZXKPMTR4AXWP7YJA, seg_pos={0 0}, upload=380680]: Done uploading piece. err: <nil> Dec 23 17:09:28.652943417 (433h40m28.524026462s) [noise=true , address=145.79.227.13:4804 , piece_id=BRMAM2YZMECAQRC7IRO57MW7HV6QY2MQ3WV6C6MN5NEYRAZV7JHA, seg_pos={0 0}, upload=380680]: Done uploading piece. err: <nil> Dec 23 17:09:28.653647879 (433h40m28.524730924s) [noise=true , address=145.79.227.37:4565 , piece_id=TJTD73TKDSXYUATA5JPKUZDBKXPCWKAIGW75QJU2W5MWYCGJYMUA, seg_pos={0 0}, upload=380680]: Done uploading piece. err: <nil> Dec 23 17:09:28.654077386 (433h40m28.525160421s) [noise=true , address=145.79.227.52:4446 , piece_id=KT7GN3Q3THGLBREEOCC4MOG5DLHFD2LLGAOMNNXBOGPWUFHMSWFA, seg_pos={0 0}, upload=380680]: Done uploading piece. err: <nil> Dec 23 17:09:28.655538408 (433h40m28.526621453s) [noise=true , address=145.79.227.42:4121 , piece_id=WTZSX53JJMAENNS2FVBBNSBGOIRNTF7HMKDP4ADYVWKNROOA32KQ, seg_pos={0 0}, upload=380680]: Done uploading piece. err: <nil> Dec 23 17:09:28.659683211 (433h40m28.530766257s) [noise=true , address=145.79.227.48:4303 , piece_id=WTOINPDQXQ44R7F23TSV3SU6SJW7ELGWU4QKSMV5I7NWNMFL4NJA, seg_pos={0 0}, upload=380680]: Done uploading piece. err: <nil> Dec 23 17:09:28.904599935 (433h40m28.77568297s) [noise=true , address=77.90.31.11:4555 , piece_id=L6TXVDKRR22ARSKVOXEU7T56YXPIUSEU5S23K23X3SP6XBGIY2LA, seg_pos={0 0}, upload=380680]: Done uploading piece. err: <nil> Dec 23 17:09:28.918645880 (433h40m28.789728925s) [noise=true , address=45.91.116.7:4009 , piece_id=7F4EFJ63IWKAB6DKBZYHQ6656T474EO2VZB45W2ILXQR7T3KTIUQ, seg_pos={0 0}, upload=380680]: Done uploading piece. err: <nil> Dec 23 17:09:28.918676428 (433h40m28.789759473s) [seg_pos={0 0}, upload=380680]: Segment reached optimal threshold of 13 pieces. Dec 23 17:09:28.918920483 (433h40m28.790003518s) [noise=true , address=45.91.116.8:4048 , piece_id=DENFYN3LEVL2PKTFM6TJBJKKDX5COUXRL5PPLACS7WVEHPJ5PYWQ, seg_pos={0 0}, upload=380680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 23 17:09:28.918928338 (433h40m28.790011373s) [noise=true , address=45.91.116.8:4048 , piece_id=DENFYN3LEVL2PKTFM6TJBJKKDX5COUXRL5PPLACS7WVEHPJ5PYWQ, seg_pos={0 0}, upload=380680]: Long tail cancellation due to optimal threshold Dec 23 17:09:28.919073044 (433h40m28.790156089s) [noise=true , address=45.91.116.5:4298 , piece_id=AZD5RY7IMH4IIQ5CNBX3WZXVUD4SOVZ34QMHTYFN63N23OQNBITQ, seg_pos={0 0}, upload=380680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 23 17:09:28.919079085 (433h40m28.79016212s) [noise=true , address=45.91.116.5:4298 , piece_id=AZD5RY7IMH4IIQ5CNBX3WZXVUD4SOVZ34QMHTYFN63N23OQNBITQ, seg_pos={0 0}, upload=380680]: Long tail cancellation due to optimal threshold Dec 23 17:09:28.919314434 (433h40m28.790397479s) [noise=true , address=145.79.227.33:4871 , piece_id=UOKKYR7PEZRFHG62ULTCA67I4C7VVDW44G7JRVLANORDQS5P5YYQ, seg_pos={0 0}, upload=380680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 23 17:09:28.919361253 (433h40m28.790444298s) [noise=true , address=145.79.227.33:4871 , piece_id=UOKKYR7PEZRFHG62ULTCA67I4C7VVDW44G7JRVLANORDQS5P5YYQ, seg_pos={0 0}, upload=380680]: Long tail cancellation due to optimal threshold Dec 23 17:09:28.919398023 (433h40m28.790481068s) [seg_pos={0 0}, upload=380680]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 17:09:28.919611008 (433h40m28.790694053s) [upload=380680]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 17:09:28.919621768 (433h40m28.790704813s) [upload=380680]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 17:09:29.220706044 (433h40m29.091789089s) [upload=380680]: Done uploading segments... Dec 23 17:09:29.220727275 (433h40m29.09181032s) [upload=380680]: Upload finished. err: <nil> Duration: 5.818259082s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X.........X.............................................X] duration: 30.003079298s trace: https://jaeger.dev.storj.io/trace/1ed3345d7ee41336 FAIL (2025-12-23 17:09:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 23 17:09:26.380966455 (433h40m26.2520495s) [upload=380680]: Starting upload Dec 23 17:09:26.381043451 (433h40m26.252126496s) [upload=380680]: Done starting upload Dec 23 17:09:26.381132912 (433h40m26.252215957s) [upload=380680]: Uploading segments... Dec 23 17:09:26.381368230 (433h40m26.252451275s) [upload=380680]: Got next segment. Inline: false Dec 23 17:09:26.381415329 (433h40m26.252498374s) [upload=380680]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 17:09:26.381421922 (433h40m26.252504977s) [upload=380680]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 17:09:26.628163610 (433h40m26.499246666s) [seg_pos={0 0}, upload=380680]: Begin upload segment... Dec 23 17:09:26.628275714 (433h40m26.499358749s) [seg_pos={0 0}, upload=380680]: Done begin upload segment. Dec 23 17:09:26.628350947 (433h40m26.499433992s) [noise=true , address=145.79.227.25:4617 , piece_id=DIQKCB3XRWK7RFEUZLGEHM3JX5LSD7PM2BHF2WSCMT3RH2P57HBQ, seg_pos={0 0}, upload=380680]: Uploading piece... Dec 23 17:09:26.628370284 (433h40m26.499453329s) [noise=true , address=145.79.227.52:4446 , piece_id=KT7GN3Q3THGLBREEOCC4MOG5DLHFD2LLGAOMNNXBOGPWUFHMSWFA, seg_pos={0 0}, upload=380680]: Uploading piece... Dec 23 17:09:26.628914840 (433h40m26.499997885s) [noise=true , address=145.79.227.7:4294 , piece_id=KIA4I2FJXUQELXOIQMNEZIOOBJTKZHUWX2H6YYJNCIFJWFDEGLCQ, seg_pos={0 0}, upload=380680]: Uploading piece... Dec 23 17:09:26.628931663 (433h40m26.500014698s) [noise=true , address=145.79.227.13:4804 , piece_id=BRMAM2YZMECAQRC7IRO57MW7HV6QY2MQ3WV6C6MN5NEYRAZV7JHA, seg_pos={0 0}, upload=380680]: Uploading piece... Dec 23 17:09:26.629249067 (433h40m26.500332102s) [noise=true , address=45.91.116.5:4298 , piece_id=AZD5RY7IMH4IIQ5CNBX3WZXVUD4SOVZ34QMHTYFN63N23OQNBITQ, seg_pos={0 0}, upload=380680]: Uploading piece... Dec 23 17:09:26.629340752 (433h40m26.500423797s) [noise=true , address=77.90.31.11:4555 , piece_id=L6TXVDKRR22ARSKVOXEU7T56YXPIUSEU5S23K23X3SP6XBGIY2LA, seg_pos={0 0}, upload=380680]: Uploading piece... Dec 23 17:09:26.629776722 (433h40m26.500859767s) [noise=true , address=145.79.227.37:4565 , piece_id=TJTD73TKDSXYUATA5JPKUZDBKXPCWKAIGW75QJU2W5MWYCGJYMUA, seg_pos={0 0}, upload=380680]: Uploading piece... Dec 23 17:09:26.630315418 (433h40m26.501398453s) [noise=true , address=145.79.227.48:4303 , piece_id=WTOINPDQXQ44R7F23TSV3SU6SJW7ELGWU4QKSMV5I7NWNMFL4NJA, seg_pos={0 0}, upload=380680]: Uploading piece... Dec 23 17:09:26.630678309 (433h40m26.501761344s) [noise=true , address=145.79.227.46:4601 , piece_id=4NNF2YLY7ZCONKV4CPPQGVLRMFEGENPZO3ZL6MLZ7XYFWJ4ZCTEQ, seg_pos={0 0}, upload=380680]: Uploading piece... Dec 23 17:09:26.630847040 (433h40m26.501930085s) [noise=true , address=145.79.227.42:4121 , piece_id=WTZSX53JJMAENNS2FVBBNSBGOIRNTF7HMKDP4ADYVWKNROOA32KQ, seg_pos={0 0}, upload=380680]: Uploading piece... Dec 23 17:09:26.631137122 (433h40m26.502220157s) [noise=true , address=145.79.227.23:4545 , piece_id=AX2MU4GIIHT7VREYS3GVHMFNLYAYEZ7ZOGWTKHA37Q7UCGR5BKZQ, seg_pos={0 0}, upload=380680]: Uploading piece... Dec 23 17:09:26.631588452 (433h40m26.502671497s) [noise=true , address=45.91.116.8:4048 , piece_id=DENFYN3LEVL2PKTFM6TJBJKKDX5COUXRL5PPLACS7WVEHPJ5PYWQ, seg_pos={0 0}, upload=380680]: Uploading piece... Dec 23 17:09:26.631766671 (433h40m26.502849716s) [noise=true , address=145.79.227.33:4871 , piece_id=UOKKYR7PEZRFHG62ULTCA67I4C7VVDW44G7JRVLANORDQS5P5YYQ, seg_pos={0 0}, upload=380680]: Uploading piece... Dec 23 17:09:26.632168226 (433h40m26.503251271s) [noise=true , address=145.79.227.43:4159 , piece_id=O6VZTNE3YU6PXK3WO7JXNC3RY5ISEZ2H4COWBVKAJ4RDLWXAW3VA, seg_pos={0 0}, upload=380680]: Uploading piece... Dec 23 17:09:26.632602562 (433h40m26.503685597s) [noise=true , address=145.79.227.11:4729 , piece_id=QDHMDK6FZ5IZIQGKCTNB7SCAPGKYYA5IUPDZZXKPMTR4AXWP7YJA, seg_pos={0 0}, upload=380680]: Uploading piece... Dec 23 17:09:26.632932121 (433h40m26.504015166s) [noise=true , address=45.91.116.7:4009 , piece_id=7F4EFJ63IWKAB6DKBZYHQ6656T474EO2VZB45W2ILXQR7T3KTIUQ, seg_pos={0 0}, upload=380680]: Uploading piece... Dec 23 17:09:28.252996980 (433h40m28.124080016s) [upload=380680]: Next returned nil segment Dec 23 17:09:28.253035976 (433h40m28.124119021s) [upload=380680]: Waiting for error group managing segments... Dec 23 17:09:28.637647661 (433h40m28.508730706s) [noise=true , address=145.79.227.46:4601 , piece_id=4NNF2YLY7ZCONKV4CPPQGVLRMFEGENPZO3ZL6MLZ7XYFWJ4ZCTEQ, seg_pos={0 0}, upload=380680]: Done uploading piece. err: <nil> Dec 23 17:09:28.646150863 (433h40m28.517233908s) [noise=true , address=145.79.227.23:4545 , piece_id=AX2MU4GIIHT7VREYS3GVHMFNLYAYEZ7ZOGWTKHA37Q7UCGR5BKZQ, seg_pos={0 0}, upload=380680]: Done uploading piece. err: <nil> Dec 23 17:09:28.651389007 (433h40m28.522472052s) [noise=true , address=145.79.227.25:4617 , piece_id=DIQKCB3XRWK7RFEUZLGEHM3JX5LSD7PM2BHF2WSCMT3RH2P57HBQ, seg_pos={0 0}, upload=380680]: Done uploading piece. err: <nil> Dec 23 17:09:28.651650015 (433h40m28.52273307s) [noise=true , address=145.79.227.43:4159 , piece_id=O6VZTNE3YU6PXK3WO7JXNC3RY5ISEZ2H4COWBVKAJ4RDLWXAW3VA, seg_pos={0 0}, upload=380680]: Done uploading piece. err: <nil> Dec 23 17:09:28.652242443 (433h40m28.523325478s) [noise=true , address=145.79.227.7:4294 , piece_id=KIA4I2FJXUQELXOIQMNEZIOOBJTKZHUWX2H6YYJNCIFJWFDEGLCQ, seg_pos={0 0}, upload=380680]: Done uploading piece. err: <nil> Dec 23 17:09:28.652411425 (433h40m28.52349446s) [noise=true , address=145.79.227.11:4729 , piece_id=QDHMDK6FZ5IZIQGKCTNB7SCAPGKYYA5IUPDZZXKPMTR4AXWP7YJA, seg_pos={0 0}, upload=380680]: Done uploading piece. err: <nil> Dec 23 17:09:28.652943417 (433h40m28.524026462s) [noise=true , address=145.79.227.13:4804 , piece_id=BRMAM2YZMECAQRC7IRO57MW7HV6QY2MQ3WV6C6MN5NEYRAZV7JHA, seg_pos={0 0}, upload=380680]: Done uploading piece. err: <nil> Dec 23 17:09:28.653647879 (433h40m28.524730924s) [noise=true , address=145.79.227.37:4565 , piece_id=TJTD73TKDSXYUATA5JPKUZDBKXPCWKAIGW75QJU2W5MWYCGJYMUA, seg_pos={0 0}, upload=380680]: Done uploading piece. err: <nil> Dec 23 17:09:28.654077386 (433h40m28.525160421s) [noise=true , address=145.79.227.52:4446 , piece_id=KT7GN3Q3THGLBREEOCC4MOG5DLHFD2LLGAOMNNXBOGPWUFHMSWFA, seg_pos={0 0}, upload=380680]: Done uploading piece. err: <nil> Dec 23 17:09:28.655538408 (433h40m28.526621453s) [noise=true , address=145.79.227.42:4121 , piece_id=WTZSX53JJMAENNS2FVBBNSBGOIRNTF7HMKDP4ADYVWKNROOA32KQ, seg_pos={0 0}, upload=380680]: Done uploading piece. err: <nil> Dec 23 17:09:28.659683211 (433h40m28.530766257s) [noise=true , address=145.79.227.48:4303 , piece_id=WTOINPDQXQ44R7F23TSV3SU6SJW7ELGWU4QKSMV5I7NWNMFL4NJA, seg_pos={0 0}, upload=380680]: Done uploading piece. err: <nil> Dec 23 17:09:28.904599935 (433h40m28.77568297s) [noise=true , address=77.90.31.11:4555 , piece_id=L6TXVDKRR22ARSKVOXEU7T56YXPIUSEU5S23K23X3SP6XBGIY2LA, seg_pos={0 0}, upload=380680]: Done uploading piece. err: <nil> Dec 23 17:09:28.918645880 (433h40m28.789728925s) [noise=true , address=45.91.116.7:4009 , piece_id=7F4EFJ63IWKAB6DKBZYHQ6656T474EO2VZB45W2ILXQR7T3KTIUQ, seg_pos={0 0}, upload=380680]: Done uploading piece. err: <nil> Dec 23 17:09:28.918676428 (433h40m28.789759473s) [seg_pos={0 0}, upload=380680]: Segment reached optimal threshold of 13 pieces. Dec 23 17:09:28.918920483 (433h40m28.790003518s) [noise=true , address=45.91.116.8:4048 , piece_id=DENFYN3LEVL2PKTFM6TJBJKKDX5COUXRL5PPLACS7WVEHPJ5PYWQ, seg_pos={0 0}, upload=380680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 23 17:09:28.918928338 (433h40m28.790011373s) [noise=true , address=45.91.116.8:4048 , piece_id=DENFYN3LEVL2PKTFM6TJBJKKDX5COUXRL5PPLACS7WVEHPJ5PYWQ, seg_pos={0 0}, upload=380680]: Long tail cancellation due to optimal threshold Dec 23 17:09:28.919073044 (433h40m28.790156089s) [noise=true , address=45.91.116.5:4298 , piece_id=AZD5RY7IMH4IIQ5CNBX3WZXVUD4SOVZ34QMHTYFN63N23OQNBITQ, seg_pos={0 0}, upload=380680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 23 17:09:28.919079085 (433h40m28.79016212s) [noise=true , address=45.91.116.5:4298 , piece_id=AZD5RY7IMH4IIQ5CNBX3WZXVUD4SOVZ34QMHTYFN63N23OQNBITQ, seg_pos={0 0}, upload=380680]: Long tail cancellation due to optimal threshold Dec 23 17:09:28.919314434 (433h40m28.790397479s) [noise=true , address=145.79.227.33:4871 , piece_id=UOKKYR7PEZRFHG62ULTCA67I4C7VVDW44G7JRVLANORDQS5P5YYQ, seg_pos={0 0}, upload=380680]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t3N84zSSMtzRBs1H47Szc2xAHwqmyr9szJahRkxMYDLto8DAx): protocol: expected piece hash; context canceled; EOF Dec 23 17:09:28.919361253 (433h40m28.790444298s) [noise=true , address=145.79.227.33:4871 , piece_id=UOKKYR7PEZRFHG62ULTCA67I4C7VVDW44G7JRVLANORDQS5P5YYQ, seg_pos={0 0}, upload=380680]: Long tail cancellation due to optimal threshold Dec 23 17:09:28.919398023 (433h40m28.790481068s) [seg_pos={0 0}, upload=380680]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 17:09:28.919611008 (433h40m28.790694053s) [upload=380680]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 17:09:28.919621768 (433h40m28.790704813s) [upload=380680]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 17:09:29.220706044 (433h40m29.091789089s) [upload=380680]: Done uploading segments... Dec 23 17:09:29.220727275 (433h40m29.09181032s) [upload=380680]: Upload finished. err: <nil> Duration: 30.003104045s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....X.........X.............................................] duration: 5.671272363s trace: https://jaeger.dev.storj.io/trace/7cce9d596d96c2fb FAIL (2025-12-23 17:08:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 23 16:33:42.217841516 (433h4m42.088924551s) [upload=380164]: Starting upload Dec 23 16:33:42.217925326 (433h4m42.089008361s) [upload=380164]: Done starting upload Dec 23 16:33:42.217975120 (433h4m42.089058155s) [upload=380164]: Uploading segments... Dec 23 16:33:42.218191021 (433h4m42.089274066s) [upload=380164]: Got next segment. Inline: false Dec 23 16:33:42.218228232 (433h4m42.089311267s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 16:33:42.218233072 (433h4m42.089316137s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 16:33:42.464012075 (433h4m42.33509511s) [seg_pos={0 0}, upload=380164]: Begin upload segment... Dec 23 16:33:42.464055629 (433h4m42.335138664s) [seg_pos={0 0}, upload=380164]: Done begin upload segment. Dec 23 16:33:42.464080826 (433h4m42.335163861s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464150278 (433h4m42.335233313s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464662895 (433h4m42.33574593s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464967405 (433h4m42.33605044s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465290650 (433h4m42.336373685s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465645145 (433h4m42.33672821s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465960345 (433h4m42.33704339s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466405773 (433h4m42.337488818s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466887240 (433h4m42.337970285s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467235243 (433h4m42.338318278s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467564109 (433h4m42.338647154s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467933462 (433h4m42.339016498s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.469791430 (433h4m42.340874465s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.470946911 (433h4m42.342029946s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.472385300 (433h4m42.343468335s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.473467610 (433h4m42.344550656s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:44.176460188 (433h4m44.047543253s) [upload=380164]: Next returned nil segment Dec 23 16:33:44.176479544 (433h4m44.047562599s) [upload=380164]: Waiting for error group managing segments... Dec 23 16:33:44.475711372 (433h4m44.346794407s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.477072674 (433h4m44.348155709s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.483581518 (433h4m44.354664553s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.489562778 (433h4m44.360645813s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.493900718 (433h4m44.364983753s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.504747682 (433h4m44.375830727s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.507440871 (433h4m44.378523916s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.726964291 (433h4m44.598047326s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.761338746 (433h4m44.632421781s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.807952966 (433h4m44.679036001s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.812606698 (433h4m44.683689733s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.965737354 (433h4m44.8368204s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373843518 (433h4m45.244926564s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373866111 (433h4m45.244949146s) [seg_pos={0 0}, upload=380164]: Segment reached optimal threshold of 13 pieces. Dec 23 16:33:45.374116078 (433h4m45.245199123s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374125225 (433h4m45.24520826s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374377926 (433h4m45.245460971s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374384488 (433h4m45.245467533s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374541508 (433h4m45.245624553s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374547108 (433h4m45.245630143s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374562777 (433h4m45.245645812s) [seg_pos={0 0}, upload=380164]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 16:33:45.374630947 (433h4m45.245713992s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 16:33:45.374637780 (433h4m45.245720815s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 16:33:45.672339675 (433h4m45.54342272s) [upload=380164]: Done uploading segments... Dec 23 16:33:45.672364471 (433h4m45.543447506s) [upload=380164]: Upload finished. err: <nil> Duration: 5.671299976s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....X.........X............................................] duration: 6.039281426s trace: https://jaeger.dev.storj.io/trace/7cce9d596d96c2fb FAIL (2025-12-23 17:07:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 23 16:33:42.217841516 (433h4m42.088924551s) [upload=380164]: Starting upload Dec 23 16:33:42.217925326 (433h4m42.089008361s) [upload=380164]: Done starting upload Dec 23 16:33:42.217975120 (433h4m42.089058155s) [upload=380164]: Uploading segments... Dec 23 16:33:42.218191021 (433h4m42.089274066s) [upload=380164]: Got next segment. Inline: false Dec 23 16:33:42.218228232 (433h4m42.089311267s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 16:33:42.218233072 (433h4m42.089316137s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 16:33:42.464012075 (433h4m42.33509511s) [seg_pos={0 0}, upload=380164]: Begin upload segment... Dec 23 16:33:42.464055629 (433h4m42.335138664s) [seg_pos={0 0}, upload=380164]: Done begin upload segment. Dec 23 16:33:42.464080826 (433h4m42.335163861s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464150278 (433h4m42.335233313s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464662895 (433h4m42.33574593s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464967405 (433h4m42.33605044s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465290650 (433h4m42.336373685s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465645145 (433h4m42.33672821s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465960345 (433h4m42.33704339s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466405773 (433h4m42.337488818s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466887240 (433h4m42.337970285s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467235243 (433h4m42.338318278s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467564109 (433h4m42.338647154s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467933462 (433h4m42.339016498s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.469791430 (433h4m42.340874465s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.470946911 (433h4m42.342029946s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.472385300 (433h4m42.343468335s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.473467610 (433h4m42.344550656s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:44.176460188 (433h4m44.047543253s) [upload=380164]: Next returned nil segment Dec 23 16:33:44.176479544 (433h4m44.047562599s) [upload=380164]: Waiting for error group managing segments... Dec 23 16:33:44.475711372 (433h4m44.346794407s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.477072674 (433h4m44.348155709s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.483581518 (433h4m44.354664553s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.489562778 (433h4m44.360645813s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.493900718 (433h4m44.364983753s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.504747682 (433h4m44.375830727s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.507440871 (433h4m44.378523916s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.726964291 (433h4m44.598047326s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.761338746 (433h4m44.632421781s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.807952966 (433h4m44.679036001s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.812606698 (433h4m44.683689733s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.965737354 (433h4m44.8368204s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373843518 (433h4m45.244926564s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373866111 (433h4m45.244949146s) [seg_pos={0 0}, upload=380164]: Segment reached optimal threshold of 13 pieces. Dec 23 16:33:45.374116078 (433h4m45.245199123s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374125225 (433h4m45.24520826s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374377926 (433h4m45.245460971s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374384488 (433h4m45.245467533s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374541508 (433h4m45.245624553s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374547108 (433h4m45.245630143s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374562777 (433h4m45.245645812s) [seg_pos={0 0}, upload=380164]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 16:33:45.374630947 (433h4m45.245713992s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 16:33:45.374637780 (433h4m45.245720815s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 16:33:45.672339675 (433h4m45.54342272s) [upload=380164]: Done uploading segments... Dec 23 16:33:45.672364471 (433h4m45.543447506s) [upload=380164]: Upload finished. err: <nil> Duration: 6.039311573s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......X.........X...........................................] duration: 5.78551457s trace: https://jaeger.dev.storj.io/trace/7cce9d596d96c2fb FAIL (2025-12-23 17:06:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 23 16:33:42.217841516 (433h4m42.088924551s) [upload=380164]: Starting upload Dec 23 16:33:42.217925326 (433h4m42.089008361s) [upload=380164]: Done starting upload Dec 23 16:33:42.217975120 (433h4m42.089058155s) [upload=380164]: Uploading segments... Dec 23 16:33:42.218191021 (433h4m42.089274066s) [upload=380164]: Got next segment. Inline: false Dec 23 16:33:42.218228232 (433h4m42.089311267s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 16:33:42.218233072 (433h4m42.089316137s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 16:33:42.464012075 (433h4m42.33509511s) [seg_pos={0 0}, upload=380164]: Begin upload segment... Dec 23 16:33:42.464055629 (433h4m42.335138664s) [seg_pos={0 0}, upload=380164]: Done begin upload segment. Dec 23 16:33:42.464080826 (433h4m42.335163861s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464150278 (433h4m42.335233313s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464662895 (433h4m42.33574593s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464967405 (433h4m42.33605044s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465290650 (433h4m42.336373685s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465645145 (433h4m42.33672821s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465960345 (433h4m42.33704339s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466405773 (433h4m42.337488818s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466887240 (433h4m42.337970285s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467235243 (433h4m42.338318278s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467564109 (433h4m42.338647154s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467933462 (433h4m42.339016498s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.469791430 (433h4m42.340874465s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.470946911 (433h4m42.342029946s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.472385300 (433h4m42.343468335s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.473467610 (433h4m42.344550656s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:44.176460188 (433h4m44.047543253s) [upload=380164]: Next returned nil segment Dec 23 16:33:44.176479544 (433h4m44.047562599s) [upload=380164]: Waiting for error group managing segments... Dec 23 16:33:44.475711372 (433h4m44.346794407s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.477072674 (433h4m44.348155709s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.483581518 (433h4m44.354664553s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.489562778 (433h4m44.360645813s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.493900718 (433h4m44.364983753s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.504747682 (433h4m44.375830727s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.507440871 (433h4m44.378523916s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.726964291 (433h4m44.598047326s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.761338746 (433h4m44.632421781s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.807952966 (433h4m44.679036001s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.812606698 (433h4m44.683689733s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.965737354 (433h4m44.8368204s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373843518 (433h4m45.244926564s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373866111 (433h4m45.244949146s) [seg_pos={0 0}, upload=380164]: Segment reached optimal threshold of 13 pieces. Dec 23 16:33:45.374116078 (433h4m45.245199123s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374125225 (433h4m45.24520826s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374377926 (433h4m45.245460971s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374384488 (433h4m45.245467533s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374541508 (433h4m45.245624553s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374547108 (433h4m45.245630143s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374562777 (433h4m45.245645812s) [seg_pos={0 0}, upload=380164]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 16:33:45.374630947 (433h4m45.245713992s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 16:33:45.374637780 (433h4m45.245720815s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 16:33:45.672339675 (433h4m45.54342272s) [upload=380164]: Done uploading segments... Dec 23 16:33:45.672364471 (433h4m45.543447506s) [upload=380164]: Upload finished. err: <nil> Duration: 5.78554617s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......X.........X..........................................] duration: 5.926146046s trace: https://jaeger.dev.storj.io/trace/7cce9d596d96c2fb FAIL (2025-12-23 17:05:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 23 16:33:42.217841516 (433h4m42.088924551s) [upload=380164]: Starting upload Dec 23 16:33:42.217925326 (433h4m42.089008361s) [upload=380164]: Done starting upload Dec 23 16:33:42.217975120 (433h4m42.089058155s) [upload=380164]: Uploading segments... Dec 23 16:33:42.218191021 (433h4m42.089274066s) [upload=380164]: Got next segment. Inline: false Dec 23 16:33:42.218228232 (433h4m42.089311267s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 16:33:42.218233072 (433h4m42.089316137s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 16:33:42.464012075 (433h4m42.33509511s) [seg_pos={0 0}, upload=380164]: Begin upload segment... Dec 23 16:33:42.464055629 (433h4m42.335138664s) [seg_pos={0 0}, upload=380164]: Done begin upload segment. Dec 23 16:33:42.464080826 (433h4m42.335163861s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464150278 (433h4m42.335233313s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464662895 (433h4m42.33574593s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464967405 (433h4m42.33605044s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465290650 (433h4m42.336373685s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465645145 (433h4m42.33672821s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465960345 (433h4m42.33704339s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466405773 (433h4m42.337488818s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466887240 (433h4m42.337970285s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467235243 (433h4m42.338318278s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467564109 (433h4m42.338647154s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467933462 (433h4m42.339016498s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.469791430 (433h4m42.340874465s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.470946911 (433h4m42.342029946s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.472385300 (433h4m42.343468335s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.473467610 (433h4m42.344550656s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:44.176460188 (433h4m44.047543253s) [upload=380164]: Next returned nil segment Dec 23 16:33:44.176479544 (433h4m44.047562599s) [upload=380164]: Waiting for error group managing segments... Dec 23 16:33:44.475711372 (433h4m44.346794407s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.477072674 (433h4m44.348155709s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.483581518 (433h4m44.354664553s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.489562778 (433h4m44.360645813s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.493900718 (433h4m44.364983753s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.504747682 (433h4m44.375830727s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.507440871 (433h4m44.378523916s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.726964291 (433h4m44.598047326s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.761338746 (433h4m44.632421781s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.807952966 (433h4m44.679036001s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.812606698 (433h4m44.683689733s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.965737354 (433h4m44.8368204s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373843518 (433h4m45.244926564s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373866111 (433h4m45.244949146s) [seg_pos={0 0}, upload=380164]: Segment reached optimal threshold of 13 pieces. Dec 23 16:33:45.374116078 (433h4m45.245199123s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374125225 (433h4m45.24520826s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374377926 (433h4m45.245460971s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374384488 (433h4m45.245467533s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374541508 (433h4m45.245624553s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374547108 (433h4m45.245630143s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374562777 (433h4m45.245645812s) [seg_pos={0 0}, upload=380164]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 16:33:45.374630947 (433h4m45.245713992s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 16:33:45.374637780 (433h4m45.245720815s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 16:33:45.672339675 (433h4m45.54342272s) [upload=380164]: Done uploading segments... Dec 23 16:33:45.672364471 (433h4m45.543447506s) [upload=380164]: Upload finished. err: <nil> Duration: 5.926177635s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........X.........X.........................................] duration: 5.986344063s trace: https://jaeger.dev.storj.io/trace/7cce9d596d96c2fb FAIL (2025-12-23 17:04:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 23 16:33:42.217841516 (433h4m42.088924551s) [upload=380164]: Starting upload Dec 23 16:33:42.217925326 (433h4m42.089008361s) [upload=380164]: Done starting upload Dec 23 16:33:42.217975120 (433h4m42.089058155s) [upload=380164]: Uploading segments... Dec 23 16:33:42.218191021 (433h4m42.089274066s) [upload=380164]: Got next segment. Inline: false Dec 23 16:33:42.218228232 (433h4m42.089311267s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 16:33:42.218233072 (433h4m42.089316137s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 16:33:42.464012075 (433h4m42.33509511s) [seg_pos={0 0}, upload=380164]: Begin upload segment... Dec 23 16:33:42.464055629 (433h4m42.335138664s) [seg_pos={0 0}, upload=380164]: Done begin upload segment. Dec 23 16:33:42.464080826 (433h4m42.335163861s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464150278 (433h4m42.335233313s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464662895 (433h4m42.33574593s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464967405 (433h4m42.33605044s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465290650 (433h4m42.336373685s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465645145 (433h4m42.33672821s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465960345 (433h4m42.33704339s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466405773 (433h4m42.337488818s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466887240 (433h4m42.337970285s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467235243 (433h4m42.338318278s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467564109 (433h4m42.338647154s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467933462 (433h4m42.339016498s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.469791430 (433h4m42.340874465s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.470946911 (433h4m42.342029946s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.472385300 (433h4m42.343468335s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.473467610 (433h4m42.344550656s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:44.176460188 (433h4m44.047543253s) [upload=380164]: Next returned nil segment Dec 23 16:33:44.176479544 (433h4m44.047562599s) [upload=380164]: Waiting for error group managing segments... Dec 23 16:33:44.475711372 (433h4m44.346794407s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.477072674 (433h4m44.348155709s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.483581518 (433h4m44.354664553s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.489562778 (433h4m44.360645813s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.493900718 (433h4m44.364983753s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.504747682 (433h4m44.375830727s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.507440871 (433h4m44.378523916s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.726964291 (433h4m44.598047326s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.761338746 (433h4m44.632421781s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.807952966 (433h4m44.679036001s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.812606698 (433h4m44.683689733s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.965737354 (433h4m44.8368204s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373843518 (433h4m45.244926564s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373866111 (433h4m45.244949146s) [seg_pos={0 0}, upload=380164]: Segment reached optimal threshold of 13 pieces. Dec 23 16:33:45.374116078 (433h4m45.245199123s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374125225 (433h4m45.24520826s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374377926 (433h4m45.245460971s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374384488 (433h4m45.245467533s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374541508 (433h4m45.245624553s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374547108 (433h4m45.245630143s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374562777 (433h4m45.245645812s) [seg_pos={0 0}, upload=380164]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 16:33:45.374630947 (433h4m45.245713992s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 16:33:45.374637780 (433h4m45.245720815s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 16:33:45.672339675 (433h4m45.54342272s) [upload=380164]: Done uploading segments... Dec 23 16:33:45.672364471 (433h4m45.543447506s) [upload=380164]: Upload finished. err: <nil> Duration: 5.986375453s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........X.........X........................................] duration: 8.207562634s trace: https://jaeger.dev.storj.io/trace/7cce9d596d96c2fb FAIL (2025-12-23 17:04:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 23 16:33:42.217841516 (433h4m42.088924551s) [upload=380164]: Starting upload Dec 23 16:33:42.217925326 (433h4m42.089008361s) [upload=380164]: Done starting upload Dec 23 16:33:42.217975120 (433h4m42.089058155s) [upload=380164]: Uploading segments... Dec 23 16:33:42.218191021 (433h4m42.089274066s) [upload=380164]: Got next segment. Inline: false Dec 23 16:33:42.218228232 (433h4m42.089311267s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 16:33:42.218233072 (433h4m42.089316137s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 16:33:42.464012075 (433h4m42.33509511s) [seg_pos={0 0}, upload=380164]: Begin upload segment... Dec 23 16:33:42.464055629 (433h4m42.335138664s) [seg_pos={0 0}, upload=380164]: Done begin upload segment. Dec 23 16:33:42.464080826 (433h4m42.335163861s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464150278 (433h4m42.335233313s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464662895 (433h4m42.33574593s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464967405 (433h4m42.33605044s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465290650 (433h4m42.336373685s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465645145 (433h4m42.33672821s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465960345 (433h4m42.33704339s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466405773 (433h4m42.337488818s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466887240 (433h4m42.337970285s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467235243 (433h4m42.338318278s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467564109 (433h4m42.338647154s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467933462 (433h4m42.339016498s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.469791430 (433h4m42.340874465s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.470946911 (433h4m42.342029946s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.472385300 (433h4m42.343468335s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.473467610 (433h4m42.344550656s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:44.176460188 (433h4m44.047543253s) [upload=380164]: Next returned nil segment Dec 23 16:33:44.176479544 (433h4m44.047562599s) [upload=380164]: Waiting for error group managing segments... Dec 23 16:33:44.475711372 (433h4m44.346794407s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.477072674 (433h4m44.348155709s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.483581518 (433h4m44.354664553s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.489562778 (433h4m44.360645813s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.493900718 (433h4m44.364983753s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.504747682 (433h4m44.375830727s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.507440871 (433h4m44.378523916s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.726964291 (433h4m44.598047326s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.761338746 (433h4m44.632421781s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.807952966 (433h4m44.679036001s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.812606698 (433h4m44.683689733s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.965737354 (433h4m44.8368204s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373843518 (433h4m45.244926564s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373866111 (433h4m45.244949146s) [seg_pos={0 0}, upload=380164]: Segment reached optimal threshold of 13 pieces. Dec 23 16:33:45.374116078 (433h4m45.245199123s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374125225 (433h4m45.24520826s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374377926 (433h4m45.245460971s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374384488 (433h4m45.245467533s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374541508 (433h4m45.245624553s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374547108 (433h4m45.245630143s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374562777 (433h4m45.245645812s) [seg_pos={0 0}, upload=380164]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 16:33:45.374630947 (433h4m45.245713992s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 16:33:45.374637780 (433h4m45.245720815s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 16:33:45.672339675 (433h4m45.54342272s) [upload=380164]: Done uploading segments... Dec 23 16:33:45.672364471 (433h4m45.543447506s) [upload=380164]: Upload finished. err: <nil> Duration: 8.20759803s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........X.........X.......................................] duration: 5.757691632s trace: https://jaeger.dev.storj.io/trace/7cce9d596d96c2fb FAIL (2025-12-23 17:02:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 23 16:33:42.217841516 (433h4m42.088924551s) [upload=380164]: Starting upload Dec 23 16:33:42.217925326 (433h4m42.089008361s) [upload=380164]: Done starting upload Dec 23 16:33:42.217975120 (433h4m42.089058155s) [upload=380164]: Uploading segments... Dec 23 16:33:42.218191021 (433h4m42.089274066s) [upload=380164]: Got next segment. Inline: false Dec 23 16:33:42.218228232 (433h4m42.089311267s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 16:33:42.218233072 (433h4m42.089316137s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 16:33:42.464012075 (433h4m42.33509511s) [seg_pos={0 0}, upload=380164]: Begin upload segment... Dec 23 16:33:42.464055629 (433h4m42.335138664s) [seg_pos={0 0}, upload=380164]: Done begin upload segment. Dec 23 16:33:42.464080826 (433h4m42.335163861s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464150278 (433h4m42.335233313s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464662895 (433h4m42.33574593s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464967405 (433h4m42.33605044s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465290650 (433h4m42.336373685s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465645145 (433h4m42.33672821s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465960345 (433h4m42.33704339s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466405773 (433h4m42.337488818s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466887240 (433h4m42.337970285s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467235243 (433h4m42.338318278s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467564109 (433h4m42.338647154s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467933462 (433h4m42.339016498s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.469791430 (433h4m42.340874465s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.470946911 (433h4m42.342029946s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.472385300 (433h4m42.343468335s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.473467610 (433h4m42.344550656s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:44.176460188 (433h4m44.047543253s) [upload=380164]: Next returned nil segment Dec 23 16:33:44.176479544 (433h4m44.047562599s) [upload=380164]: Waiting for error group managing segments... Dec 23 16:33:44.475711372 (433h4m44.346794407s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.477072674 (433h4m44.348155709s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.483581518 (433h4m44.354664553s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.489562778 (433h4m44.360645813s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.493900718 (433h4m44.364983753s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.504747682 (433h4m44.375830727s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.507440871 (433h4m44.378523916s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.726964291 (433h4m44.598047326s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.761338746 (433h4m44.632421781s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.807952966 (433h4m44.679036001s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.812606698 (433h4m44.683689733s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.965737354 (433h4m44.8368204s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373843518 (433h4m45.244926564s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373866111 (433h4m45.244949146s) [seg_pos={0 0}, upload=380164]: Segment reached optimal threshold of 13 pieces. Dec 23 16:33:45.374116078 (433h4m45.245199123s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374125225 (433h4m45.24520826s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374377926 (433h4m45.245460971s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374384488 (433h4m45.245467533s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374541508 (433h4m45.245624553s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374547108 (433h4m45.245630143s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374562777 (433h4m45.245645812s) [seg_pos={0 0}, upload=380164]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 16:33:45.374630947 (433h4m45.245713992s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 16:33:45.374637780 (433h4m45.245720815s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 16:33:45.672339675 (433h4m45.54342272s) [upload=380164]: Done uploading segments... Dec 23 16:33:45.672364471 (433h4m45.543447506s) [upload=380164]: Upload finished. err: <nil> Duration: 5.757719184s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........X.........X......................................] duration: 5.916602022s trace: https://jaeger.dev.storj.io/trace/7cce9d596d96c2fb FAIL (2025-12-23 17:02:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 23 16:33:42.217841516 (433h4m42.088924551s) [upload=380164]: Starting upload Dec 23 16:33:42.217925326 (433h4m42.089008361s) [upload=380164]: Done starting upload Dec 23 16:33:42.217975120 (433h4m42.089058155s) [upload=380164]: Uploading segments... Dec 23 16:33:42.218191021 (433h4m42.089274066s) [upload=380164]: Got next segment. Inline: false Dec 23 16:33:42.218228232 (433h4m42.089311267s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 16:33:42.218233072 (433h4m42.089316137s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 16:33:42.464012075 (433h4m42.33509511s) [seg_pos={0 0}, upload=380164]: Begin upload segment... Dec 23 16:33:42.464055629 (433h4m42.335138664s) [seg_pos={0 0}, upload=380164]: Done begin upload segment. Dec 23 16:33:42.464080826 (433h4m42.335163861s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464150278 (433h4m42.335233313s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464662895 (433h4m42.33574593s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464967405 (433h4m42.33605044s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465290650 (433h4m42.336373685s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465645145 (433h4m42.33672821s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465960345 (433h4m42.33704339s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466405773 (433h4m42.337488818s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466887240 (433h4m42.337970285s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467235243 (433h4m42.338318278s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467564109 (433h4m42.338647154s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467933462 (433h4m42.339016498s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.469791430 (433h4m42.340874465s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.470946911 (433h4m42.342029946s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.472385300 (433h4m42.343468335s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.473467610 (433h4m42.344550656s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:44.176460188 (433h4m44.047543253s) [upload=380164]: Next returned nil segment Dec 23 16:33:44.176479544 (433h4m44.047562599s) [upload=380164]: Waiting for error group managing segments... Dec 23 16:33:44.475711372 (433h4m44.346794407s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.477072674 (433h4m44.348155709s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.483581518 (433h4m44.354664553s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.489562778 (433h4m44.360645813s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.493900718 (433h4m44.364983753s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.504747682 (433h4m44.375830727s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.507440871 (433h4m44.378523916s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.726964291 (433h4m44.598047326s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.761338746 (433h4m44.632421781s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.807952966 (433h4m44.679036001s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.812606698 (433h4m44.683689733s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.965737354 (433h4m44.8368204s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373843518 (433h4m45.244926564s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373866111 (433h4m45.244949146s) [seg_pos={0 0}, upload=380164]: Segment reached optimal threshold of 13 pieces. Dec 23 16:33:45.374116078 (433h4m45.245199123s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374125225 (433h4m45.24520826s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374377926 (433h4m45.245460971s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374384488 (433h4m45.245467533s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374541508 (433h4m45.245624553s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374547108 (433h4m45.245630143s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374562777 (433h4m45.245645812s) [seg_pos={0 0}, upload=380164]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 16:33:45.374630947 (433h4m45.245713992s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 16:33:45.374637780 (433h4m45.245720815s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 16:33:45.672339675 (433h4m45.54342272s) [upload=380164]: Done uploading segments... Dec 23 16:33:45.672364471 (433h4m45.543447506s) [upload=380164]: Upload finished. err: <nil> Duration: 5.916626479s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............X.........X.....................................] duration: 5.945527263s trace: https://jaeger.dev.storj.io/trace/7cce9d596d96c2fb FAIL (2025-12-23 17:01:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 23 16:33:42.217841516 (433h4m42.088924551s) [upload=380164]: Starting upload Dec 23 16:33:42.217925326 (433h4m42.089008361s) [upload=380164]: Done starting upload Dec 23 16:33:42.217975120 (433h4m42.089058155s) [upload=380164]: Uploading segments... Dec 23 16:33:42.218191021 (433h4m42.089274066s) [upload=380164]: Got next segment. Inline: false Dec 23 16:33:42.218228232 (433h4m42.089311267s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 16:33:42.218233072 (433h4m42.089316137s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 16:33:42.464012075 (433h4m42.33509511s) [seg_pos={0 0}, upload=380164]: Begin upload segment... Dec 23 16:33:42.464055629 (433h4m42.335138664s) [seg_pos={0 0}, upload=380164]: Done begin upload segment. Dec 23 16:33:42.464080826 (433h4m42.335163861s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464150278 (433h4m42.335233313s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464662895 (433h4m42.33574593s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464967405 (433h4m42.33605044s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465290650 (433h4m42.336373685s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465645145 (433h4m42.33672821s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465960345 (433h4m42.33704339s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466405773 (433h4m42.337488818s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466887240 (433h4m42.337970285s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467235243 (433h4m42.338318278s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467564109 (433h4m42.338647154s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467933462 (433h4m42.339016498s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.469791430 (433h4m42.340874465s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.470946911 (433h4m42.342029946s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.472385300 (433h4m42.343468335s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.473467610 (433h4m42.344550656s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:44.176460188 (433h4m44.047543253s) [upload=380164]: Next returned nil segment Dec 23 16:33:44.176479544 (433h4m44.047562599s) [upload=380164]: Waiting for error group managing segments... Dec 23 16:33:44.475711372 (433h4m44.346794407s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.477072674 (433h4m44.348155709s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.483581518 (433h4m44.354664553s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.489562778 (433h4m44.360645813s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.493900718 (433h4m44.364983753s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.504747682 (433h4m44.375830727s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.507440871 (433h4m44.378523916s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.726964291 (433h4m44.598047326s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.761338746 (433h4m44.632421781s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.807952966 (433h4m44.679036001s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.812606698 (433h4m44.683689733s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.965737354 (433h4m44.8368204s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373843518 (433h4m45.244926564s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373866111 (433h4m45.244949146s) [seg_pos={0 0}, upload=380164]: Segment reached optimal threshold of 13 pieces. Dec 23 16:33:45.374116078 (433h4m45.245199123s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374125225 (433h4m45.24520826s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374377926 (433h4m45.245460971s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374384488 (433h4m45.245467533s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374541508 (433h4m45.245624553s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374547108 (433h4m45.245630143s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374562777 (433h4m45.245645812s) [seg_pos={0 0}, upload=380164]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 16:33:45.374630947 (433h4m45.245713992s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 16:33:45.374637780 (433h4m45.245720815s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 16:33:45.672339675 (433h4m45.54342272s) [upload=380164]: Done uploading segments... Dec 23 16:33:45.672364471 (433h4m45.543447506s) [upload=380164]: Upload finished. err: <nil> Duration: 5.945555807s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............X.........X....................................] duration: 6.231214478s trace: https://jaeger.dev.storj.io/trace/7cce9d596d96c2fb FAIL (2025-12-23 17:00:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 23 16:33:42.217841516 (433h4m42.088924551s) [upload=380164]: Starting upload Dec 23 16:33:42.217925326 (433h4m42.089008361s) [upload=380164]: Done starting upload Dec 23 16:33:42.217975120 (433h4m42.089058155s) [upload=380164]: Uploading segments... Dec 23 16:33:42.218191021 (433h4m42.089274066s) [upload=380164]: Got next segment. Inline: false Dec 23 16:33:42.218228232 (433h4m42.089311267s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 16:33:42.218233072 (433h4m42.089316137s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 16:33:42.464012075 (433h4m42.33509511s) [seg_pos={0 0}, upload=380164]: Begin upload segment... Dec 23 16:33:42.464055629 (433h4m42.335138664s) [seg_pos={0 0}, upload=380164]: Done begin upload segment. Dec 23 16:33:42.464080826 (433h4m42.335163861s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464150278 (433h4m42.335233313s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464662895 (433h4m42.33574593s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464967405 (433h4m42.33605044s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465290650 (433h4m42.336373685s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465645145 (433h4m42.33672821s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465960345 (433h4m42.33704339s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466405773 (433h4m42.337488818s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466887240 (433h4m42.337970285s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467235243 (433h4m42.338318278s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467564109 (433h4m42.338647154s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467933462 (433h4m42.339016498s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.469791430 (433h4m42.340874465s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.470946911 (433h4m42.342029946s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.472385300 (433h4m42.343468335s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.473467610 (433h4m42.344550656s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:44.176460188 (433h4m44.047543253s) [upload=380164]: Next returned nil segment Dec 23 16:33:44.176479544 (433h4m44.047562599s) [upload=380164]: Waiting for error group managing segments... Dec 23 16:33:44.475711372 (433h4m44.346794407s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.477072674 (433h4m44.348155709s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.483581518 (433h4m44.354664553s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.489562778 (433h4m44.360645813s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.493900718 (433h4m44.364983753s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.504747682 (433h4m44.375830727s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.507440871 (433h4m44.378523916s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.726964291 (433h4m44.598047326s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.761338746 (433h4m44.632421781s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.807952966 (433h4m44.679036001s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.812606698 (433h4m44.683689733s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.965737354 (433h4m44.8368204s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373843518 (433h4m45.244926564s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373866111 (433h4m45.244949146s) [seg_pos={0 0}, upload=380164]: Segment reached optimal threshold of 13 pieces. Dec 23 16:33:45.374116078 (433h4m45.245199123s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374125225 (433h4m45.24520826s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374377926 (433h4m45.245460971s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374384488 (433h4m45.245467533s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374541508 (433h4m45.245624553s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374547108 (433h4m45.245630143s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374562777 (433h4m45.245645812s) [seg_pos={0 0}, upload=380164]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 16:33:45.374630947 (433h4m45.245713992s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 16:33:45.374637780 (433h4m45.245720815s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 16:33:45.672339675 (433h4m45.54342272s) [upload=380164]: Done uploading segments... Dec 23 16:33:45.672364471 (433h4m45.543447506s) [upload=380164]: Upload finished. err: <nil> Duration: 6.231260085s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............X.........X...................................] duration: 20.610202661s trace: https://jaeger.dev.storj.io/trace/7cce9d596d96c2fb FAIL (2025-12-23 16:59:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 23 16:33:42.217841516 (433h4m42.088924551s) [upload=380164]: Starting upload Dec 23 16:33:42.217925326 (433h4m42.089008361s) [upload=380164]: Done starting upload Dec 23 16:33:42.217975120 (433h4m42.089058155s) [upload=380164]: Uploading segments... Dec 23 16:33:42.218191021 (433h4m42.089274066s) [upload=380164]: Got next segment. Inline: false Dec 23 16:33:42.218228232 (433h4m42.089311267s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 16:33:42.218233072 (433h4m42.089316137s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 16:33:42.464012075 (433h4m42.33509511s) [seg_pos={0 0}, upload=380164]: Begin upload segment... Dec 23 16:33:42.464055629 (433h4m42.335138664s) [seg_pos={0 0}, upload=380164]: Done begin upload segment. Dec 23 16:33:42.464080826 (433h4m42.335163861s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464150278 (433h4m42.335233313s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464662895 (433h4m42.33574593s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464967405 (433h4m42.33605044s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465290650 (433h4m42.336373685s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465645145 (433h4m42.33672821s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465960345 (433h4m42.33704339s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466405773 (433h4m42.337488818s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466887240 (433h4m42.337970285s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467235243 (433h4m42.338318278s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467564109 (433h4m42.338647154s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467933462 (433h4m42.339016498s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.469791430 (433h4m42.340874465s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.470946911 (433h4m42.342029946s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.472385300 (433h4m42.343468335s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.473467610 (433h4m42.344550656s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:44.176460188 (433h4m44.047543253s) [upload=380164]: Next returned nil segment Dec 23 16:33:44.176479544 (433h4m44.047562599s) [upload=380164]: Waiting for error group managing segments... Dec 23 16:33:44.475711372 (433h4m44.346794407s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.477072674 (433h4m44.348155709s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.483581518 (433h4m44.354664553s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.489562778 (433h4m44.360645813s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.493900718 (433h4m44.364983753s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.504747682 (433h4m44.375830727s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.507440871 (433h4m44.378523916s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.726964291 (433h4m44.598047326s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.761338746 (433h4m44.632421781s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.807952966 (433h4m44.679036001s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.812606698 (433h4m44.683689733s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.965737354 (433h4m44.8368204s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373843518 (433h4m45.244926564s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373866111 (433h4m45.244949146s) [seg_pos={0 0}, upload=380164]: Segment reached optimal threshold of 13 pieces. Dec 23 16:33:45.374116078 (433h4m45.245199123s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374125225 (433h4m45.24520826s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374377926 (433h4m45.245460971s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374384488 (433h4m45.245467533s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374541508 (433h4m45.245624553s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374547108 (433h4m45.245630143s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374562777 (433h4m45.245645812s) [seg_pos={0 0}, upload=380164]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 16:33:45.374630947 (433h4m45.245713992s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 16:33:45.374637780 (433h4m45.245720815s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 16:33:45.672339675 (433h4m45.54342272s) [upload=380164]: Done uploading segments... Dec 23 16:33:45.672364471 (433h4m45.543447506s) [upload=380164]: Upload finished. err: <nil> Duration: 20.610262905s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............X.........X..................................] duration: 5.669472487s trace: https://jaeger.dev.storj.io/trace/7cce9d596d96c2fb FAIL (2025-12-23 16:58:26 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 23 16:33:42.217841516 (433h4m42.088924551s) [upload=380164]: Starting upload Dec 23 16:33:42.217925326 (433h4m42.089008361s) [upload=380164]: Done starting upload Dec 23 16:33:42.217975120 (433h4m42.089058155s) [upload=380164]: Uploading segments... Dec 23 16:33:42.218191021 (433h4m42.089274066s) [upload=380164]: Got next segment. Inline: false Dec 23 16:33:42.218228232 (433h4m42.089311267s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 16:33:42.218233072 (433h4m42.089316137s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 16:33:42.464012075 (433h4m42.33509511s) [seg_pos={0 0}, upload=380164]: Begin upload segment... Dec 23 16:33:42.464055629 (433h4m42.335138664s) [seg_pos={0 0}, upload=380164]: Done begin upload segment. Dec 23 16:33:42.464080826 (433h4m42.335163861s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464150278 (433h4m42.335233313s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464662895 (433h4m42.33574593s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464967405 (433h4m42.33605044s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465290650 (433h4m42.336373685s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465645145 (433h4m42.33672821s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465960345 (433h4m42.33704339s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466405773 (433h4m42.337488818s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466887240 (433h4m42.337970285s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467235243 (433h4m42.338318278s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467564109 (433h4m42.338647154s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467933462 (433h4m42.339016498s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.469791430 (433h4m42.340874465s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.470946911 (433h4m42.342029946s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.472385300 (433h4m42.343468335s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.473467610 (433h4m42.344550656s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:44.176460188 (433h4m44.047543253s) [upload=380164]: Next returned nil segment Dec 23 16:33:44.176479544 (433h4m44.047562599s) [upload=380164]: Waiting for error group managing segments... Dec 23 16:33:44.475711372 (433h4m44.346794407s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.477072674 (433h4m44.348155709s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.483581518 (433h4m44.354664553s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.489562778 (433h4m44.360645813s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.493900718 (433h4m44.364983753s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.504747682 (433h4m44.375830727s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.507440871 (433h4m44.378523916s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.726964291 (433h4m44.598047326s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.761338746 (433h4m44.632421781s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.807952966 (433h4m44.679036001s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.812606698 (433h4m44.683689733s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.965737354 (433h4m44.8368204s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373843518 (433h4m45.244926564s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373866111 (433h4m45.244949146s) [seg_pos={0 0}, upload=380164]: Segment reached optimal threshold of 13 pieces. Dec 23 16:33:45.374116078 (433h4m45.245199123s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374125225 (433h4m45.24520826s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374377926 (433h4m45.245460971s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374384488 (433h4m45.245467533s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374541508 (433h4m45.245624553s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374547108 (433h4m45.245630143s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374562777 (433h4m45.245645812s) [seg_pos={0 0}, upload=380164]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 16:33:45.374630947 (433h4m45.245713992s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 16:33:45.374637780 (433h4m45.245720815s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 16:33:45.672339675 (433h4m45.54342272s) [upload=380164]: Done uploading segments... Dec 23 16:33:45.672364471 (433h4m45.543447506s) [upload=380164]: Upload finished. err: <nil> Duration: 5.669501623s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................X.........X.................................] duration: 5.939714183s trace: https://jaeger.dev.storj.io/trace/7cce9d596d96c2fb FAIL (2025-12-23 16:58:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 23 16:33:42.217841516 (433h4m42.088924551s) [upload=380164]: Starting upload Dec 23 16:33:42.217925326 (433h4m42.089008361s) [upload=380164]: Done starting upload Dec 23 16:33:42.217975120 (433h4m42.089058155s) [upload=380164]: Uploading segments... Dec 23 16:33:42.218191021 (433h4m42.089274066s) [upload=380164]: Got next segment. Inline: false Dec 23 16:33:42.218228232 (433h4m42.089311267s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 16:33:42.218233072 (433h4m42.089316137s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 16:33:42.464012075 (433h4m42.33509511s) [seg_pos={0 0}, upload=380164]: Begin upload segment... Dec 23 16:33:42.464055629 (433h4m42.335138664s) [seg_pos={0 0}, upload=380164]: Done begin upload segment. Dec 23 16:33:42.464080826 (433h4m42.335163861s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464150278 (433h4m42.335233313s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464662895 (433h4m42.33574593s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464967405 (433h4m42.33605044s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465290650 (433h4m42.336373685s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465645145 (433h4m42.33672821s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465960345 (433h4m42.33704339s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466405773 (433h4m42.337488818s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466887240 (433h4m42.337970285s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467235243 (433h4m42.338318278s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467564109 (433h4m42.338647154s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467933462 (433h4m42.339016498s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.469791430 (433h4m42.340874465s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.470946911 (433h4m42.342029946s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.472385300 (433h4m42.343468335s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.473467610 (433h4m42.344550656s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:44.176460188 (433h4m44.047543253s) [upload=380164]: Next returned nil segment Dec 23 16:33:44.176479544 (433h4m44.047562599s) [upload=380164]: Waiting for error group managing segments... Dec 23 16:33:44.475711372 (433h4m44.346794407s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.477072674 (433h4m44.348155709s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.483581518 (433h4m44.354664553s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.489562778 (433h4m44.360645813s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.493900718 (433h4m44.364983753s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.504747682 (433h4m44.375830727s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.507440871 (433h4m44.378523916s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.726964291 (433h4m44.598047326s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.761338746 (433h4m44.632421781s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.807952966 (433h4m44.679036001s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.812606698 (433h4m44.683689733s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.965737354 (433h4m44.8368204s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373843518 (433h4m45.244926564s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373866111 (433h4m45.244949146s) [seg_pos={0 0}, upload=380164]: Segment reached optimal threshold of 13 pieces. Dec 23 16:33:45.374116078 (433h4m45.245199123s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374125225 (433h4m45.24520826s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374377926 (433h4m45.245460971s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374384488 (433h4m45.245467533s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374541508 (433h4m45.245624553s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374547108 (433h4m45.245630143s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374562777 (433h4m45.245645812s) [seg_pos={0 0}, upload=380164]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 16:33:45.374630947 (433h4m45.245713992s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 16:33:45.374637780 (433h4m45.245720815s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 16:33:45.672339675 (433h4m45.54342272s) [upload=380164]: Done uploading segments... Dec 23 16:33:45.672364471 (433h4m45.543447506s) [upload=380164]: Upload finished. err: <nil> Duration: 5.939757786s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................X.........X................................] duration: 18.284182321s trace: https://jaeger.dev.storj.io/trace/7cce9d596d96c2fb FAIL (2025-12-23 16:57:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 23 16:33:42.217841516 (433h4m42.088924551s) [upload=380164]: Starting upload Dec 23 16:33:42.217925326 (433h4m42.089008361s) [upload=380164]: Done starting upload Dec 23 16:33:42.217975120 (433h4m42.089058155s) [upload=380164]: Uploading segments... Dec 23 16:33:42.218191021 (433h4m42.089274066s) [upload=380164]: Got next segment. Inline: false Dec 23 16:33:42.218228232 (433h4m42.089311267s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 16:33:42.218233072 (433h4m42.089316137s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 16:33:42.464012075 (433h4m42.33509511s) [seg_pos={0 0}, upload=380164]: Begin upload segment... Dec 23 16:33:42.464055629 (433h4m42.335138664s) [seg_pos={0 0}, upload=380164]: Done begin upload segment. Dec 23 16:33:42.464080826 (433h4m42.335163861s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464150278 (433h4m42.335233313s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464662895 (433h4m42.33574593s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464967405 (433h4m42.33605044s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465290650 (433h4m42.336373685s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465645145 (433h4m42.33672821s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465960345 (433h4m42.33704339s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466405773 (433h4m42.337488818s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466887240 (433h4m42.337970285s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467235243 (433h4m42.338318278s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467564109 (433h4m42.338647154s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467933462 (433h4m42.339016498s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.469791430 (433h4m42.340874465s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.470946911 (433h4m42.342029946s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.472385300 (433h4m42.343468335s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.473467610 (433h4m42.344550656s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:44.176460188 (433h4m44.047543253s) [upload=380164]: Next returned nil segment Dec 23 16:33:44.176479544 (433h4m44.047562599s) [upload=380164]: Waiting for error group managing segments... Dec 23 16:33:44.475711372 (433h4m44.346794407s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.477072674 (433h4m44.348155709s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.483581518 (433h4m44.354664553s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.489562778 (433h4m44.360645813s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.493900718 (433h4m44.364983753s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.504747682 (433h4m44.375830727s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.507440871 (433h4m44.378523916s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.726964291 (433h4m44.598047326s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.761338746 (433h4m44.632421781s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.807952966 (433h4m44.679036001s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.812606698 (433h4m44.683689733s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.965737354 (433h4m44.8368204s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373843518 (433h4m45.244926564s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373866111 (433h4m45.244949146s) [seg_pos={0 0}, upload=380164]: Segment reached optimal threshold of 13 pieces. Dec 23 16:33:45.374116078 (433h4m45.245199123s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374125225 (433h4m45.24520826s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374377926 (433h4m45.245460971s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374384488 (433h4m45.245467533s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374541508 (433h4m45.245624553s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374547108 (433h4m45.245630143s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374562777 (433h4m45.245645812s) [seg_pos={0 0}, upload=380164]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 16:33:45.374630947 (433h4m45.245713992s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 16:33:45.374637780 (433h4m45.245720815s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 16:33:45.672339675 (433h4m45.54342272s) [upload=380164]: Done uploading segments... Dec 23 16:33:45.672364471 (433h4m45.543447506s) [upload=380164]: Upload finished. err: <nil> Duration: 18.284239631s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................X.........X...............................] duration: 5.754875387s trace: https://jaeger.dev.storj.io/trace/7cce9d596d96c2fb FAIL (2025-12-23 16:56:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 23 16:33:42.217841516 (433h4m42.088924551s) [upload=380164]: Starting upload Dec 23 16:33:42.217925326 (433h4m42.089008361s) [upload=380164]: Done starting upload Dec 23 16:33:42.217975120 (433h4m42.089058155s) [upload=380164]: Uploading segments... Dec 23 16:33:42.218191021 (433h4m42.089274066s) [upload=380164]: Got next segment. Inline: false Dec 23 16:33:42.218228232 (433h4m42.089311267s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 16:33:42.218233072 (433h4m42.089316137s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 16:33:42.464012075 (433h4m42.33509511s) [seg_pos={0 0}, upload=380164]: Begin upload segment... Dec 23 16:33:42.464055629 (433h4m42.335138664s) [seg_pos={0 0}, upload=380164]: Done begin upload segment. Dec 23 16:33:42.464080826 (433h4m42.335163861s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464150278 (433h4m42.335233313s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464662895 (433h4m42.33574593s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464967405 (433h4m42.33605044s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465290650 (433h4m42.336373685s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465645145 (433h4m42.33672821s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465960345 (433h4m42.33704339s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466405773 (433h4m42.337488818s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466887240 (433h4m42.337970285s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467235243 (433h4m42.338318278s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467564109 (433h4m42.338647154s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467933462 (433h4m42.339016498s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.469791430 (433h4m42.340874465s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.470946911 (433h4m42.342029946s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.472385300 (433h4m42.343468335s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.473467610 (433h4m42.344550656s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:44.176460188 (433h4m44.047543253s) [upload=380164]: Next returned nil segment Dec 23 16:33:44.176479544 (433h4m44.047562599s) [upload=380164]: Waiting for error group managing segments... Dec 23 16:33:44.475711372 (433h4m44.346794407s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.477072674 (433h4m44.348155709s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.483581518 (433h4m44.354664553s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.489562778 (433h4m44.360645813s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.493900718 (433h4m44.364983753s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.504747682 (433h4m44.375830727s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.507440871 (433h4m44.378523916s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.726964291 (433h4m44.598047326s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.761338746 (433h4m44.632421781s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.807952966 (433h4m44.679036001s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.812606698 (433h4m44.683689733s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.965737354 (433h4m44.8368204s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373843518 (433h4m45.244926564s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373866111 (433h4m45.244949146s) [seg_pos={0 0}, upload=380164]: Segment reached optimal threshold of 13 pieces. Dec 23 16:33:45.374116078 (433h4m45.245199123s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374125225 (433h4m45.24520826s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374377926 (433h4m45.245460971s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374384488 (433h4m45.245467533s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374541508 (433h4m45.245624553s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374547108 (433h4m45.245630143s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374562777 (433h4m45.245645812s) [seg_pos={0 0}, upload=380164]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 16:33:45.374630947 (433h4m45.245713992s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 16:33:45.374637780 (433h4m45.245720815s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 16:33:45.672339675 (433h4m45.54342272s) [upload=380164]: Done uploading segments... Dec 23 16:33:45.672364471 (433h4m45.543447506s) [upload=380164]: Upload finished. err: <nil> Duration: 5.754904043s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................X.........X..............................] duration: 5.770985003s trace: https://jaeger.dev.storj.io/trace/7cce9d596d96c2fb FAIL (2025-12-23 16:55:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 23 16:33:42.217841516 (433h4m42.088924551s) [upload=380164]: Starting upload Dec 23 16:33:42.217925326 (433h4m42.089008361s) [upload=380164]: Done starting upload Dec 23 16:33:42.217975120 (433h4m42.089058155s) [upload=380164]: Uploading segments... Dec 23 16:33:42.218191021 (433h4m42.089274066s) [upload=380164]: Got next segment. Inline: false Dec 23 16:33:42.218228232 (433h4m42.089311267s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 16:33:42.218233072 (433h4m42.089316137s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 16:33:42.464012075 (433h4m42.33509511s) [seg_pos={0 0}, upload=380164]: Begin upload segment... Dec 23 16:33:42.464055629 (433h4m42.335138664s) [seg_pos={0 0}, upload=380164]: Done begin upload segment. Dec 23 16:33:42.464080826 (433h4m42.335163861s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464150278 (433h4m42.335233313s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464662895 (433h4m42.33574593s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464967405 (433h4m42.33605044s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465290650 (433h4m42.336373685s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465645145 (433h4m42.33672821s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465960345 (433h4m42.33704339s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466405773 (433h4m42.337488818s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466887240 (433h4m42.337970285s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467235243 (433h4m42.338318278s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467564109 (433h4m42.338647154s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467933462 (433h4m42.339016498s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.469791430 (433h4m42.340874465s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.470946911 (433h4m42.342029946s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.472385300 (433h4m42.343468335s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.473467610 (433h4m42.344550656s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:44.176460188 (433h4m44.047543253s) [upload=380164]: Next returned nil segment Dec 23 16:33:44.176479544 (433h4m44.047562599s) [upload=380164]: Waiting for error group managing segments... Dec 23 16:33:44.475711372 (433h4m44.346794407s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.477072674 (433h4m44.348155709s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.483581518 (433h4m44.354664553s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.489562778 (433h4m44.360645813s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.493900718 (433h4m44.364983753s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.504747682 (433h4m44.375830727s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.507440871 (433h4m44.378523916s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.726964291 (433h4m44.598047326s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.761338746 (433h4m44.632421781s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.807952966 (433h4m44.679036001s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.812606698 (433h4m44.683689733s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.965737354 (433h4m44.8368204s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373843518 (433h4m45.244926564s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373866111 (433h4m45.244949146s) [seg_pos={0 0}, upload=380164]: Segment reached optimal threshold of 13 pieces. Dec 23 16:33:45.374116078 (433h4m45.245199123s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374125225 (433h4m45.24520826s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374377926 (433h4m45.245460971s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374384488 (433h4m45.245467533s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374541508 (433h4m45.245624553s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374547108 (433h4m45.245630143s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374562777 (433h4m45.245645812s) [seg_pos={0 0}, upload=380164]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 16:33:45.374630947 (433h4m45.245713992s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 16:33:45.374637780 (433h4m45.245720815s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 16:33:45.672339675 (433h4m45.54342272s) [upload=380164]: Done uploading segments... Dec 23 16:33:45.672364471 (433h4m45.543447506s) [upload=380164]: Upload finished. err: <nil> Duration: 5.771017806s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................X.........X.............................] duration: 5.910939801s trace: https://jaeger.dev.storj.io/trace/7cce9d596d96c2fb FAIL (2025-12-23 16:54:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 23 16:33:42.217841516 (433h4m42.088924551s) [upload=380164]: Starting upload Dec 23 16:33:42.217925326 (433h4m42.089008361s) [upload=380164]: Done starting upload Dec 23 16:33:42.217975120 (433h4m42.089058155s) [upload=380164]: Uploading segments... Dec 23 16:33:42.218191021 (433h4m42.089274066s) [upload=380164]: Got next segment. Inline: false Dec 23 16:33:42.218228232 (433h4m42.089311267s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 16:33:42.218233072 (433h4m42.089316137s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 16:33:42.464012075 (433h4m42.33509511s) [seg_pos={0 0}, upload=380164]: Begin upload segment... Dec 23 16:33:42.464055629 (433h4m42.335138664s) [seg_pos={0 0}, upload=380164]: Done begin upload segment. Dec 23 16:33:42.464080826 (433h4m42.335163861s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464150278 (433h4m42.335233313s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464662895 (433h4m42.33574593s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464967405 (433h4m42.33605044s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465290650 (433h4m42.336373685s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465645145 (433h4m42.33672821s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465960345 (433h4m42.33704339s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466405773 (433h4m42.337488818s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466887240 (433h4m42.337970285s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467235243 (433h4m42.338318278s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467564109 (433h4m42.338647154s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467933462 (433h4m42.339016498s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.469791430 (433h4m42.340874465s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.470946911 (433h4m42.342029946s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.472385300 (433h4m42.343468335s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.473467610 (433h4m42.344550656s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:44.176460188 (433h4m44.047543253s) [upload=380164]: Next returned nil segment Dec 23 16:33:44.176479544 (433h4m44.047562599s) [upload=380164]: Waiting for error group managing segments... Dec 23 16:33:44.475711372 (433h4m44.346794407s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.477072674 (433h4m44.348155709s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.483581518 (433h4m44.354664553s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.489562778 (433h4m44.360645813s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.493900718 (433h4m44.364983753s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.504747682 (433h4m44.375830727s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.507440871 (433h4m44.378523916s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.726964291 (433h4m44.598047326s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.761338746 (433h4m44.632421781s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.807952966 (433h4m44.679036001s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.812606698 (433h4m44.683689733s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.965737354 (433h4m44.8368204s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373843518 (433h4m45.244926564s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373866111 (433h4m45.244949146s) [seg_pos={0 0}, upload=380164]: Segment reached optimal threshold of 13 pieces. Dec 23 16:33:45.374116078 (433h4m45.245199123s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374125225 (433h4m45.24520826s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374377926 (433h4m45.245460971s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374384488 (433h4m45.245467533s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374541508 (433h4m45.245624553s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374547108 (433h4m45.245630143s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374562777 (433h4m45.245645812s) [seg_pos={0 0}, upload=380164]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 16:33:45.374630947 (433h4m45.245713992s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 16:33:45.374637780 (433h4m45.245720815s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 16:33:45.672339675 (433h4m45.54342272s) [upload=380164]: Done uploading segments... Dec 23 16:33:45.672364471 (433h4m45.543447506s) [upload=380164]: Upload finished. err: <nil> Duration: 5.910957695s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................X.........X............................] duration: 26.604554335s trace: https://jaeger.dev.storj.io/trace/7cce9d596d96c2fb FAIL (2025-12-23 16:54:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 23 16:33:42.217841516 (433h4m42.088924551s) [upload=380164]: Starting upload Dec 23 16:33:42.217925326 (433h4m42.089008361s) [upload=380164]: Done starting upload Dec 23 16:33:42.217975120 (433h4m42.089058155s) [upload=380164]: Uploading segments... Dec 23 16:33:42.218191021 (433h4m42.089274066s) [upload=380164]: Got next segment. Inline: false Dec 23 16:33:42.218228232 (433h4m42.089311267s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 16:33:42.218233072 (433h4m42.089316137s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 16:33:42.464012075 (433h4m42.33509511s) [seg_pos={0 0}, upload=380164]: Begin upload segment... Dec 23 16:33:42.464055629 (433h4m42.335138664s) [seg_pos={0 0}, upload=380164]: Done begin upload segment. Dec 23 16:33:42.464080826 (433h4m42.335163861s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464150278 (433h4m42.335233313s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464662895 (433h4m42.33574593s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464967405 (433h4m42.33605044s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465290650 (433h4m42.336373685s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465645145 (433h4m42.33672821s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465960345 (433h4m42.33704339s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466405773 (433h4m42.337488818s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466887240 (433h4m42.337970285s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467235243 (433h4m42.338318278s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467564109 (433h4m42.338647154s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467933462 (433h4m42.339016498s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.469791430 (433h4m42.340874465s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.470946911 (433h4m42.342029946s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.472385300 (433h4m42.343468335s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.473467610 (433h4m42.344550656s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:44.176460188 (433h4m44.047543253s) [upload=380164]: Next returned nil segment Dec 23 16:33:44.176479544 (433h4m44.047562599s) [upload=380164]: Waiting for error group managing segments... Dec 23 16:33:44.475711372 (433h4m44.346794407s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.477072674 (433h4m44.348155709s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.483581518 (433h4m44.354664553s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.489562778 (433h4m44.360645813s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.493900718 (433h4m44.364983753s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.504747682 (433h4m44.375830727s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.507440871 (433h4m44.378523916s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.726964291 (433h4m44.598047326s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.761338746 (433h4m44.632421781s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.807952966 (433h4m44.679036001s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.812606698 (433h4m44.683689733s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.965737354 (433h4m44.8368204s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373843518 (433h4m45.244926564s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373866111 (433h4m45.244949146s) [seg_pos={0 0}, upload=380164]: Segment reached optimal threshold of 13 pieces. Dec 23 16:33:45.374116078 (433h4m45.245199123s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374125225 (433h4m45.24520826s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374377926 (433h4m45.245460971s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374384488 (433h4m45.245467533s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374541508 (433h4m45.245624553s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374547108 (433h4m45.245630143s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374562777 (433h4m45.245645812s) [seg_pos={0 0}, upload=380164]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 16:33:45.374630947 (433h4m45.245713992s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 16:33:45.374637780 (433h4m45.245720815s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 16:33:45.672339675 (433h4m45.54342272s) [upload=380164]: Done uploading segments... Dec 23 16:33:45.672364471 (433h4m45.543447506s) [upload=380164]: Upload finished. err: <nil> Duration: 26.604573392s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................X.........X...........................] duration: 24.171982445s trace: https://jaeger.dev.storj.io/trace/7cce9d596d96c2fb FAIL (2025-12-23 16:53:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 23 16:33:42.217841516 (433h4m42.088924551s) [upload=380164]: Starting upload Dec 23 16:33:42.217925326 (433h4m42.089008361s) [upload=380164]: Done starting upload Dec 23 16:33:42.217975120 (433h4m42.089058155s) [upload=380164]: Uploading segments... Dec 23 16:33:42.218191021 (433h4m42.089274066s) [upload=380164]: Got next segment. Inline: false Dec 23 16:33:42.218228232 (433h4m42.089311267s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 16:33:42.218233072 (433h4m42.089316137s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 16:33:42.464012075 (433h4m42.33509511s) [seg_pos={0 0}, upload=380164]: Begin upload segment... Dec 23 16:33:42.464055629 (433h4m42.335138664s) [seg_pos={0 0}, upload=380164]: Done begin upload segment. Dec 23 16:33:42.464080826 (433h4m42.335163861s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464150278 (433h4m42.335233313s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464662895 (433h4m42.33574593s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464967405 (433h4m42.33605044s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465290650 (433h4m42.336373685s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465645145 (433h4m42.33672821s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465960345 (433h4m42.33704339s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466405773 (433h4m42.337488818s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466887240 (433h4m42.337970285s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467235243 (433h4m42.338318278s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467564109 (433h4m42.338647154s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467933462 (433h4m42.339016498s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.469791430 (433h4m42.340874465s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.470946911 (433h4m42.342029946s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.472385300 (433h4m42.343468335s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.473467610 (433h4m42.344550656s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:44.176460188 (433h4m44.047543253s) [upload=380164]: Next returned nil segment Dec 23 16:33:44.176479544 (433h4m44.047562599s) [upload=380164]: Waiting for error group managing segments... Dec 23 16:33:44.475711372 (433h4m44.346794407s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.477072674 (433h4m44.348155709s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.483581518 (433h4m44.354664553s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.489562778 (433h4m44.360645813s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.493900718 (433h4m44.364983753s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.504747682 (433h4m44.375830727s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.507440871 (433h4m44.378523916s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.726964291 (433h4m44.598047326s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.761338746 (433h4m44.632421781s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.807952966 (433h4m44.679036001s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.812606698 (433h4m44.683689733s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.965737354 (433h4m44.8368204s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373843518 (433h4m45.244926564s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373866111 (433h4m45.244949146s) [seg_pos={0 0}, upload=380164]: Segment reached optimal threshold of 13 pieces. Dec 23 16:33:45.374116078 (433h4m45.245199123s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374125225 (433h4m45.24520826s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374377926 (433h4m45.245460971s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374384488 (433h4m45.245467533s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374541508 (433h4m45.245624553s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374547108 (433h4m45.245630143s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374562777 (433h4m45.245645812s) [seg_pos={0 0}, upload=380164]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 16:33:45.374630947 (433h4m45.245713992s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 16:33:45.374637780 (433h4m45.245720815s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 16:33:45.672339675 (433h4m45.54342272s) [upload=380164]: Done uploading segments... Dec 23 16:33:45.672364471 (433h4m45.543447506s) [upload=380164]: Upload finished. err: <nil> Duration: 24.172010068s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................X.........X..........................] duration: 5.7615787s trace: https://jaeger.dev.storj.io/trace/7cce9d596d96c2fb FAIL (2025-12-23 16:53:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 23 16:33:42.217841516 (433h4m42.088924551s) [upload=380164]: Starting upload Dec 23 16:33:42.217925326 (433h4m42.089008361s) [upload=380164]: Done starting upload Dec 23 16:33:42.217975120 (433h4m42.089058155s) [upload=380164]: Uploading segments... Dec 23 16:33:42.218191021 (433h4m42.089274066s) [upload=380164]: Got next segment. Inline: false Dec 23 16:33:42.218228232 (433h4m42.089311267s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 16:33:42.218233072 (433h4m42.089316137s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 16:33:42.464012075 (433h4m42.33509511s) [seg_pos={0 0}, upload=380164]: Begin upload segment... Dec 23 16:33:42.464055629 (433h4m42.335138664s) [seg_pos={0 0}, upload=380164]: Done begin upload segment. Dec 23 16:33:42.464080826 (433h4m42.335163861s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464150278 (433h4m42.335233313s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464662895 (433h4m42.33574593s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464967405 (433h4m42.33605044s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465290650 (433h4m42.336373685s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465645145 (433h4m42.33672821s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465960345 (433h4m42.33704339s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466405773 (433h4m42.337488818s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466887240 (433h4m42.337970285s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467235243 (433h4m42.338318278s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467564109 (433h4m42.338647154s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467933462 (433h4m42.339016498s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.469791430 (433h4m42.340874465s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.470946911 (433h4m42.342029946s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.472385300 (433h4m42.343468335s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.473467610 (433h4m42.344550656s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:44.176460188 (433h4m44.047543253s) [upload=380164]: Next returned nil segment Dec 23 16:33:44.176479544 (433h4m44.047562599s) [upload=380164]: Waiting for error group managing segments... Dec 23 16:33:44.475711372 (433h4m44.346794407s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.477072674 (433h4m44.348155709s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.483581518 (433h4m44.354664553s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.489562778 (433h4m44.360645813s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.493900718 (433h4m44.364983753s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.504747682 (433h4m44.375830727s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.507440871 (433h4m44.378523916s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.726964291 (433h4m44.598047326s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.761338746 (433h4m44.632421781s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.807952966 (433h4m44.679036001s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.812606698 (433h4m44.683689733s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.965737354 (433h4m44.8368204s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373843518 (433h4m45.244926564s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373866111 (433h4m45.244949146s) [seg_pos={0 0}, upload=380164]: Segment reached optimal threshold of 13 pieces. Dec 23 16:33:45.374116078 (433h4m45.245199123s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374125225 (433h4m45.24520826s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374377926 (433h4m45.245460971s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374384488 (433h4m45.245467533s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374541508 (433h4m45.245624553s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374547108 (433h4m45.245630143s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374562777 (433h4m45.245645812s) [seg_pos={0 0}, upload=380164]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 16:33:45.374630947 (433h4m45.245713992s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 16:33:45.374637780 (433h4m45.245720815s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 16:33:45.672339675 (433h4m45.54342272s) [upload=380164]: Done uploading segments... Dec 23 16:33:45.672364471 (433h4m45.543447506s) [upload=380164]: Upload finished. err: <nil> Duration: 5.761609258s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [X.......................X.........X.........................] duration: 5.67732087s trace: https://jaeger.dev.storj.io/trace/7cce9d596d96c2fb FAIL (2025-12-23 16:52:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 23 16:33:42.217841516 (433h4m42.088924551s) [upload=380164]: Starting upload Dec 23 16:33:42.217925326 (433h4m42.089008361s) [upload=380164]: Done starting upload Dec 23 16:33:42.217975120 (433h4m42.089058155s) [upload=380164]: Uploading segments... Dec 23 16:33:42.218191021 (433h4m42.089274066s) [upload=380164]: Got next segment. Inline: false Dec 23 16:33:42.218228232 (433h4m42.089311267s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 16:33:42.218233072 (433h4m42.089316137s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 16:33:42.464012075 (433h4m42.33509511s) [seg_pos={0 0}, upload=380164]: Begin upload segment... Dec 23 16:33:42.464055629 (433h4m42.335138664s) [seg_pos={0 0}, upload=380164]: Done begin upload segment. Dec 23 16:33:42.464080826 (433h4m42.335163861s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464150278 (433h4m42.335233313s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464662895 (433h4m42.33574593s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464967405 (433h4m42.33605044s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465290650 (433h4m42.336373685s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465645145 (433h4m42.33672821s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465960345 (433h4m42.33704339s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466405773 (433h4m42.337488818s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466887240 (433h4m42.337970285s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467235243 (433h4m42.338318278s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467564109 (433h4m42.338647154s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467933462 (433h4m42.339016498s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.469791430 (433h4m42.340874465s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.470946911 (433h4m42.342029946s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.472385300 (433h4m42.343468335s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.473467610 (433h4m42.344550656s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:44.176460188 (433h4m44.047543253s) [upload=380164]: Next returned nil segment Dec 23 16:33:44.176479544 (433h4m44.047562599s) [upload=380164]: Waiting for error group managing segments... Dec 23 16:33:44.475711372 (433h4m44.346794407s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.477072674 (433h4m44.348155709s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.483581518 (433h4m44.354664553s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.489562778 (433h4m44.360645813s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.493900718 (433h4m44.364983753s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.504747682 (433h4m44.375830727s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.507440871 (433h4m44.378523916s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.726964291 (433h4m44.598047326s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.761338746 (433h4m44.632421781s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.807952966 (433h4m44.679036001s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.812606698 (433h4m44.683689733s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.965737354 (433h4m44.8368204s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373843518 (433h4m45.244926564s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373866111 (433h4m45.244949146s) [seg_pos={0 0}, upload=380164]: Segment reached optimal threshold of 13 pieces. Dec 23 16:33:45.374116078 (433h4m45.245199123s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374125225 (433h4m45.24520826s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374377926 (433h4m45.245460971s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374384488 (433h4m45.245467533s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374541508 (433h4m45.245624553s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374547108 (433h4m45.245630143s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374562777 (433h4m45.245645812s) [seg_pos={0 0}, upload=380164]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 16:33:45.374630947 (433h4m45.245713992s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 16:33:45.374637780 (433h4m45.245720815s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 16:33:45.672339675 (433h4m45.54342272s) [upload=380164]: Done uploading segments... Dec 23 16:33:45.672364471 (433h4m45.543447506s) [upload=380164]: Upload finished. err: <nil> Duration: 5.677352151s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X.......................X.........X........................] duration: 5.922630494s trace: https://jaeger.dev.storj.io/trace/7cce9d596d96c2fb FAIL (2025-12-23 16:51:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 23 16:33:42.217841516 (433h4m42.088924551s) [upload=380164]: Starting upload Dec 23 16:33:42.217925326 (433h4m42.089008361s) [upload=380164]: Done starting upload Dec 23 16:33:42.217975120 (433h4m42.089058155s) [upload=380164]: Uploading segments... Dec 23 16:33:42.218191021 (433h4m42.089274066s) [upload=380164]: Got next segment. Inline: false Dec 23 16:33:42.218228232 (433h4m42.089311267s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 16:33:42.218233072 (433h4m42.089316137s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 16:33:42.464012075 (433h4m42.33509511s) [seg_pos={0 0}, upload=380164]: Begin upload segment... Dec 23 16:33:42.464055629 (433h4m42.335138664s) [seg_pos={0 0}, upload=380164]: Done begin upload segment. Dec 23 16:33:42.464080826 (433h4m42.335163861s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464150278 (433h4m42.335233313s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464662895 (433h4m42.33574593s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464967405 (433h4m42.33605044s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465290650 (433h4m42.336373685s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465645145 (433h4m42.33672821s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465960345 (433h4m42.33704339s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466405773 (433h4m42.337488818s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466887240 (433h4m42.337970285s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467235243 (433h4m42.338318278s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467564109 (433h4m42.338647154s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467933462 (433h4m42.339016498s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.469791430 (433h4m42.340874465s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.470946911 (433h4m42.342029946s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.472385300 (433h4m42.343468335s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.473467610 (433h4m42.344550656s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:44.176460188 (433h4m44.047543253s) [upload=380164]: Next returned nil segment Dec 23 16:33:44.176479544 (433h4m44.047562599s) [upload=380164]: Waiting for error group managing segments... Dec 23 16:33:44.475711372 (433h4m44.346794407s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.477072674 (433h4m44.348155709s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.483581518 (433h4m44.354664553s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.489562778 (433h4m44.360645813s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.493900718 (433h4m44.364983753s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.504747682 (433h4m44.375830727s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.507440871 (433h4m44.378523916s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.726964291 (433h4m44.598047326s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.761338746 (433h4m44.632421781s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.807952966 (433h4m44.679036001s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.812606698 (433h4m44.683689733s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.965737354 (433h4m44.8368204s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373843518 (433h4m45.244926564s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373866111 (433h4m45.244949146s) [seg_pos={0 0}, upload=380164]: Segment reached optimal threshold of 13 pieces. Dec 23 16:33:45.374116078 (433h4m45.245199123s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374125225 (433h4m45.24520826s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374377926 (433h4m45.245460971s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374384488 (433h4m45.245467533s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374541508 (433h4m45.245624553s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374547108 (433h4m45.245630143s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374562777 (433h4m45.245645812s) [seg_pos={0 0}, upload=380164]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 16:33:45.374630947 (433h4m45.245713992s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 16:33:45.374637780 (433h4m45.245720815s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 16:33:45.672339675 (433h4m45.54342272s) [upload=380164]: Done uploading segments... Dec 23 16:33:45.672364471 (433h4m45.543447506s) [upload=380164]: Upload finished. err: <nil> Duration: 5.922650631s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..X.......................X.........X.......................] duration: 5.679203406s trace: https://jaeger.dev.storj.io/trace/7cce9d596d96c2fb FAIL (2025-12-23 16:51:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 23 16:33:42.217841516 (433h4m42.088924551s) [upload=380164]: Starting upload Dec 23 16:33:42.217925326 (433h4m42.089008361s) [upload=380164]: Done starting upload Dec 23 16:33:42.217975120 (433h4m42.089058155s) [upload=380164]: Uploading segments... Dec 23 16:33:42.218191021 (433h4m42.089274066s) [upload=380164]: Got next segment. Inline: false Dec 23 16:33:42.218228232 (433h4m42.089311267s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 16:33:42.218233072 (433h4m42.089316137s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 16:33:42.464012075 (433h4m42.33509511s) [seg_pos={0 0}, upload=380164]: Begin upload segment... Dec 23 16:33:42.464055629 (433h4m42.335138664s) [seg_pos={0 0}, upload=380164]: Done begin upload segment. Dec 23 16:33:42.464080826 (433h4m42.335163861s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464150278 (433h4m42.335233313s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464662895 (433h4m42.33574593s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464967405 (433h4m42.33605044s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465290650 (433h4m42.336373685s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465645145 (433h4m42.33672821s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465960345 (433h4m42.33704339s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466405773 (433h4m42.337488818s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466887240 (433h4m42.337970285s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467235243 (433h4m42.338318278s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467564109 (433h4m42.338647154s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467933462 (433h4m42.339016498s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.469791430 (433h4m42.340874465s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.470946911 (433h4m42.342029946s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.472385300 (433h4m42.343468335s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.473467610 (433h4m42.344550656s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:44.176460188 (433h4m44.047543253s) [upload=380164]: Next returned nil segment Dec 23 16:33:44.176479544 (433h4m44.047562599s) [upload=380164]: Waiting for error group managing segments... Dec 23 16:33:44.475711372 (433h4m44.346794407s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.477072674 (433h4m44.348155709s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.483581518 (433h4m44.354664553s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.489562778 (433h4m44.360645813s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.493900718 (433h4m44.364983753s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.504747682 (433h4m44.375830727s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.507440871 (433h4m44.378523916s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.726964291 (433h4m44.598047326s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.761338746 (433h4m44.632421781s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.807952966 (433h4m44.679036001s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.812606698 (433h4m44.683689733s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.965737354 (433h4m44.8368204s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373843518 (433h4m45.244926564s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373866111 (433h4m45.244949146s) [seg_pos={0 0}, upload=380164]: Segment reached optimal threshold of 13 pieces. Dec 23 16:33:45.374116078 (433h4m45.245199123s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374125225 (433h4m45.24520826s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374377926 (433h4m45.245460971s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374384488 (433h4m45.245467533s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374541508 (433h4m45.245624553s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374547108 (433h4m45.245630143s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374562777 (433h4m45.245645812s) [seg_pos={0 0}, upload=380164]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 16:33:45.374630947 (433h4m45.245713992s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 16:33:45.374637780 (433h4m45.245720815s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 16:33:45.672339675 (433h4m45.54342272s) [upload=380164]: Done uploading segments... Dec 23 16:33:45.672364471 (433h4m45.543447506s) [upload=380164]: Upload finished. err: <nil> Duration: 5.679223444s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...X.......................X.........X......................] duration: 5.676792505s trace: https://jaeger.dev.storj.io/trace/7cce9d596d96c2fb FAIL (2025-12-23 16:50:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 23 16:33:42.217841516 (433h4m42.088924551s) [upload=380164]: Starting upload Dec 23 16:33:42.217925326 (433h4m42.089008361s) [upload=380164]: Done starting upload Dec 23 16:33:42.217975120 (433h4m42.089058155s) [upload=380164]: Uploading segments... Dec 23 16:33:42.218191021 (433h4m42.089274066s) [upload=380164]: Got next segment. Inline: false Dec 23 16:33:42.218228232 (433h4m42.089311267s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 16:33:42.218233072 (433h4m42.089316137s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 16:33:42.464012075 (433h4m42.33509511s) [seg_pos={0 0}, upload=380164]: Begin upload segment... Dec 23 16:33:42.464055629 (433h4m42.335138664s) [seg_pos={0 0}, upload=380164]: Done begin upload segment. Dec 23 16:33:42.464080826 (433h4m42.335163861s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464150278 (433h4m42.335233313s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464662895 (433h4m42.33574593s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464967405 (433h4m42.33605044s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465290650 (433h4m42.336373685s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465645145 (433h4m42.33672821s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465960345 (433h4m42.33704339s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466405773 (433h4m42.337488818s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466887240 (433h4m42.337970285s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467235243 (433h4m42.338318278s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467564109 (433h4m42.338647154s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467933462 (433h4m42.339016498s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.469791430 (433h4m42.340874465s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.470946911 (433h4m42.342029946s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.472385300 (433h4m42.343468335s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.473467610 (433h4m42.344550656s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:44.176460188 (433h4m44.047543253s) [upload=380164]: Next returned nil segment Dec 23 16:33:44.176479544 (433h4m44.047562599s) [upload=380164]: Waiting for error group managing segments... Dec 23 16:33:44.475711372 (433h4m44.346794407s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.477072674 (433h4m44.348155709s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.483581518 (433h4m44.354664553s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.489562778 (433h4m44.360645813s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.493900718 (433h4m44.364983753s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.504747682 (433h4m44.375830727s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.507440871 (433h4m44.378523916s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.726964291 (433h4m44.598047326s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.761338746 (433h4m44.632421781s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.807952966 (433h4m44.679036001s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.812606698 (433h4m44.683689733s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.965737354 (433h4m44.8368204s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373843518 (433h4m45.244926564s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373866111 (433h4m45.244949146s) [seg_pos={0 0}, upload=380164]: Segment reached optimal threshold of 13 pieces. Dec 23 16:33:45.374116078 (433h4m45.245199123s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374125225 (433h4m45.24520826s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374377926 (433h4m45.245460971s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374384488 (433h4m45.245467533s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374541508 (433h4m45.245624553s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374547108 (433h4m45.245630143s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374562777 (433h4m45.245645812s) [seg_pos={0 0}, upload=380164]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 16:33:45.374630947 (433h4m45.245713992s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 16:33:45.374637780 (433h4m45.245720815s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 16:33:45.672339675 (433h4m45.54342272s) [upload=380164]: Done uploading segments... Dec 23 16:33:45.672364471 (433h4m45.543447506s) [upload=380164]: Upload finished. err: <nil> Duration: 5.676823323s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....X.......................X.........X.....................] duration: 5.626052978s trace: https://jaeger.dev.storj.io/trace/7cce9d596d96c2fb FAIL (2025-12-23 16:50:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 23 16:33:42.217841516 (433h4m42.088924551s) [upload=380164]: Starting upload Dec 23 16:33:42.217925326 (433h4m42.089008361s) [upload=380164]: Done starting upload Dec 23 16:33:42.217975120 (433h4m42.089058155s) [upload=380164]: Uploading segments... Dec 23 16:33:42.218191021 (433h4m42.089274066s) [upload=380164]: Got next segment. Inline: false Dec 23 16:33:42.218228232 (433h4m42.089311267s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 16:33:42.218233072 (433h4m42.089316137s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 16:33:42.464012075 (433h4m42.33509511s) [seg_pos={0 0}, upload=380164]: Begin upload segment... Dec 23 16:33:42.464055629 (433h4m42.335138664s) [seg_pos={0 0}, upload=380164]: Done begin upload segment. Dec 23 16:33:42.464080826 (433h4m42.335163861s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464150278 (433h4m42.335233313s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464662895 (433h4m42.33574593s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464967405 (433h4m42.33605044s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465290650 (433h4m42.336373685s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465645145 (433h4m42.33672821s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465960345 (433h4m42.33704339s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466405773 (433h4m42.337488818s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466887240 (433h4m42.337970285s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467235243 (433h4m42.338318278s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467564109 (433h4m42.338647154s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467933462 (433h4m42.339016498s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.469791430 (433h4m42.340874465s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.470946911 (433h4m42.342029946s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.472385300 (433h4m42.343468335s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.473467610 (433h4m42.344550656s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:44.176460188 (433h4m44.047543253s) [upload=380164]: Next returned nil segment Dec 23 16:33:44.176479544 (433h4m44.047562599s) [upload=380164]: Waiting for error group managing segments... Dec 23 16:33:44.475711372 (433h4m44.346794407s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.477072674 (433h4m44.348155709s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.483581518 (433h4m44.354664553s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.489562778 (433h4m44.360645813s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.493900718 (433h4m44.364983753s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.504747682 (433h4m44.375830727s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.507440871 (433h4m44.378523916s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.726964291 (433h4m44.598047326s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.761338746 (433h4m44.632421781s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.807952966 (433h4m44.679036001s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.812606698 (433h4m44.683689733s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.965737354 (433h4m44.8368204s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373843518 (433h4m45.244926564s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373866111 (433h4m45.244949146s) [seg_pos={0 0}, upload=380164]: Segment reached optimal threshold of 13 pieces. Dec 23 16:33:45.374116078 (433h4m45.245199123s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374125225 (433h4m45.24520826s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374377926 (433h4m45.245460971s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374384488 (433h4m45.245467533s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374541508 (433h4m45.245624553s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374547108 (433h4m45.245630143s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374562777 (433h4m45.245645812s) [seg_pos={0 0}, upload=380164]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 16:33:45.374630947 (433h4m45.245713992s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 16:33:45.374637780 (433h4m45.245720815s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 16:33:45.672339675 (433h4m45.54342272s) [upload=380164]: Done uploading segments... Dec 23 16:33:45.672364471 (433h4m45.543447506s) [upload=380164]: Upload finished. err: <nil> Duration: 5.626082273s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....X.......................X.........X....................] duration: 5.945538974s trace: https://jaeger.dev.storj.io/trace/7cce9d596d96c2fb FAIL (2025-12-23 16:50:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 23 16:33:42.217841516 (433h4m42.088924551s) [upload=380164]: Starting upload Dec 23 16:33:42.217925326 (433h4m42.089008361s) [upload=380164]: Done starting upload Dec 23 16:33:42.217975120 (433h4m42.089058155s) [upload=380164]: Uploading segments... Dec 23 16:33:42.218191021 (433h4m42.089274066s) [upload=380164]: Got next segment. Inline: false Dec 23 16:33:42.218228232 (433h4m42.089311267s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 16:33:42.218233072 (433h4m42.089316137s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 16:33:42.464012075 (433h4m42.33509511s) [seg_pos={0 0}, upload=380164]: Begin upload segment... Dec 23 16:33:42.464055629 (433h4m42.335138664s) [seg_pos={0 0}, upload=380164]: Done begin upload segment. Dec 23 16:33:42.464080826 (433h4m42.335163861s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464150278 (433h4m42.335233313s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464662895 (433h4m42.33574593s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464967405 (433h4m42.33605044s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465290650 (433h4m42.336373685s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465645145 (433h4m42.33672821s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465960345 (433h4m42.33704339s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466405773 (433h4m42.337488818s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466887240 (433h4m42.337970285s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467235243 (433h4m42.338318278s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467564109 (433h4m42.338647154s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467933462 (433h4m42.339016498s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.469791430 (433h4m42.340874465s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.470946911 (433h4m42.342029946s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.472385300 (433h4m42.343468335s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.473467610 (433h4m42.344550656s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:44.176460188 (433h4m44.047543253s) [upload=380164]: Next returned nil segment Dec 23 16:33:44.176479544 (433h4m44.047562599s) [upload=380164]: Waiting for error group managing segments... Dec 23 16:33:44.475711372 (433h4m44.346794407s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.477072674 (433h4m44.348155709s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.483581518 (433h4m44.354664553s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.489562778 (433h4m44.360645813s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.493900718 (433h4m44.364983753s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.504747682 (433h4m44.375830727s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.507440871 (433h4m44.378523916s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.726964291 (433h4m44.598047326s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.761338746 (433h4m44.632421781s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.807952966 (433h4m44.679036001s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.812606698 (433h4m44.683689733s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.965737354 (433h4m44.8368204s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373843518 (433h4m45.244926564s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373866111 (433h4m45.244949146s) [seg_pos={0 0}, upload=380164]: Segment reached optimal threshold of 13 pieces. Dec 23 16:33:45.374116078 (433h4m45.245199123s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374125225 (433h4m45.24520826s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374377926 (433h4m45.245460971s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374384488 (433h4m45.245467533s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374541508 (433h4m45.245624553s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374547108 (433h4m45.245630143s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374562777 (433h4m45.245645812s) [seg_pos={0 0}, upload=380164]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 16:33:45.374630947 (433h4m45.245713992s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 16:33:45.374637780 (433h4m45.245720815s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 16:33:45.672339675 (433h4m45.54342272s) [upload=380164]: Done uploading segments... Dec 23 16:33:45.672364471 (433h4m45.543447506s) [upload=380164]: Upload finished. err: <nil> Duration: 5.945570953s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......X.......................X.........X...................] duration: 5.684585717s trace: https://jaeger.dev.storj.io/trace/7cce9d596d96c2fb FAIL (2025-12-23 16:48:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 23 16:33:42.217841516 (433h4m42.088924551s) [upload=380164]: Starting upload Dec 23 16:33:42.217925326 (433h4m42.089008361s) [upload=380164]: Done starting upload Dec 23 16:33:42.217975120 (433h4m42.089058155s) [upload=380164]: Uploading segments... Dec 23 16:33:42.218191021 (433h4m42.089274066s) [upload=380164]: Got next segment. Inline: false Dec 23 16:33:42.218228232 (433h4m42.089311267s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 16:33:42.218233072 (433h4m42.089316137s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 16:33:42.464012075 (433h4m42.33509511s) [seg_pos={0 0}, upload=380164]: Begin upload segment... Dec 23 16:33:42.464055629 (433h4m42.335138664s) [seg_pos={0 0}, upload=380164]: Done begin upload segment. Dec 23 16:33:42.464080826 (433h4m42.335163861s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464150278 (433h4m42.335233313s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464662895 (433h4m42.33574593s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464967405 (433h4m42.33605044s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465290650 (433h4m42.336373685s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465645145 (433h4m42.33672821s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465960345 (433h4m42.33704339s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466405773 (433h4m42.337488818s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466887240 (433h4m42.337970285s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467235243 (433h4m42.338318278s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467564109 (433h4m42.338647154s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467933462 (433h4m42.339016498s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.469791430 (433h4m42.340874465s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.470946911 (433h4m42.342029946s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.472385300 (433h4m42.343468335s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.473467610 (433h4m42.344550656s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:44.176460188 (433h4m44.047543253s) [upload=380164]: Next returned nil segment Dec 23 16:33:44.176479544 (433h4m44.047562599s) [upload=380164]: Waiting for error group managing segments... Dec 23 16:33:44.475711372 (433h4m44.346794407s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.477072674 (433h4m44.348155709s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.483581518 (433h4m44.354664553s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.489562778 (433h4m44.360645813s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.493900718 (433h4m44.364983753s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.504747682 (433h4m44.375830727s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.507440871 (433h4m44.378523916s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.726964291 (433h4m44.598047326s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.761338746 (433h4m44.632421781s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.807952966 (433h4m44.679036001s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.812606698 (433h4m44.683689733s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.965737354 (433h4m44.8368204s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373843518 (433h4m45.244926564s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373866111 (433h4m45.244949146s) [seg_pos={0 0}, upload=380164]: Segment reached optimal threshold of 13 pieces. Dec 23 16:33:45.374116078 (433h4m45.245199123s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374125225 (433h4m45.24520826s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374377926 (433h4m45.245460971s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374384488 (433h4m45.245467533s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374541508 (433h4m45.245624553s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374547108 (433h4m45.245630143s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374562777 (433h4m45.245645812s) [seg_pos={0 0}, upload=380164]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 16:33:45.374630947 (433h4m45.245713992s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 16:33:45.374637780 (433h4m45.245720815s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 16:33:45.672339675 (433h4m45.54342272s) [upload=380164]: Done uploading segments... Dec 23 16:33:45.672364471 (433h4m45.543447506s) [upload=380164]: Upload finished. err: <nil> Duration: 5.684614452s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......X.......................X.........X..................] duration: 5.954608282s trace: https://jaeger.dev.storj.io/trace/7cce9d596d96c2fb FAIL (2025-12-23 16:47:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 23 16:33:42.217841516 (433h4m42.088924551s) [upload=380164]: Starting upload Dec 23 16:33:42.217925326 (433h4m42.089008361s) [upload=380164]: Done starting upload Dec 23 16:33:42.217975120 (433h4m42.089058155s) [upload=380164]: Uploading segments... Dec 23 16:33:42.218191021 (433h4m42.089274066s) [upload=380164]: Got next segment. Inline: false Dec 23 16:33:42.218228232 (433h4m42.089311267s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 16:33:42.218233072 (433h4m42.089316137s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 16:33:42.464012075 (433h4m42.33509511s) [seg_pos={0 0}, upload=380164]: Begin upload segment... Dec 23 16:33:42.464055629 (433h4m42.335138664s) [seg_pos={0 0}, upload=380164]: Done begin upload segment. Dec 23 16:33:42.464080826 (433h4m42.335163861s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464150278 (433h4m42.335233313s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464662895 (433h4m42.33574593s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464967405 (433h4m42.33605044s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465290650 (433h4m42.336373685s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465645145 (433h4m42.33672821s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465960345 (433h4m42.33704339s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466405773 (433h4m42.337488818s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466887240 (433h4m42.337970285s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467235243 (433h4m42.338318278s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467564109 (433h4m42.338647154s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467933462 (433h4m42.339016498s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.469791430 (433h4m42.340874465s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.470946911 (433h4m42.342029946s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.472385300 (433h4m42.343468335s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.473467610 (433h4m42.344550656s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:44.176460188 (433h4m44.047543253s) [upload=380164]: Next returned nil segment Dec 23 16:33:44.176479544 (433h4m44.047562599s) [upload=380164]: Waiting for error group managing segments... Dec 23 16:33:44.475711372 (433h4m44.346794407s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.477072674 (433h4m44.348155709s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.483581518 (433h4m44.354664553s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.489562778 (433h4m44.360645813s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.493900718 (433h4m44.364983753s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.504747682 (433h4m44.375830727s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.507440871 (433h4m44.378523916s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.726964291 (433h4m44.598047326s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.761338746 (433h4m44.632421781s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.807952966 (433h4m44.679036001s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.812606698 (433h4m44.683689733s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.965737354 (433h4m44.8368204s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373843518 (433h4m45.244926564s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373866111 (433h4m45.244949146s) [seg_pos={0 0}, upload=380164]: Segment reached optimal threshold of 13 pieces. Dec 23 16:33:45.374116078 (433h4m45.245199123s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374125225 (433h4m45.24520826s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374377926 (433h4m45.245460971s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374384488 (433h4m45.245467533s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374541508 (433h4m45.245624553s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374547108 (433h4m45.245630143s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374562777 (433h4m45.245645812s) [seg_pos={0 0}, upload=380164]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 16:33:45.374630947 (433h4m45.245713992s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 16:33:45.374637780 (433h4m45.245720815s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 16:33:45.672339675 (433h4m45.54342272s) [upload=380164]: Done uploading segments... Dec 23 16:33:45.672364471 (433h4m45.543447506s) [upload=380164]: Upload finished. err: <nil> Duration: 5.954647756s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........X.......................X.........X.................] duration: 5.651282981s trace: https://jaeger.dev.storj.io/trace/7cce9d596d96c2fb FAIL (2025-12-23 16:47:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 23 16:33:42.217841516 (433h4m42.088924551s) [upload=380164]: Starting upload Dec 23 16:33:42.217925326 (433h4m42.089008361s) [upload=380164]: Done starting upload Dec 23 16:33:42.217975120 (433h4m42.089058155s) [upload=380164]: Uploading segments... Dec 23 16:33:42.218191021 (433h4m42.089274066s) [upload=380164]: Got next segment. Inline: false Dec 23 16:33:42.218228232 (433h4m42.089311267s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 16:33:42.218233072 (433h4m42.089316137s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 16:33:42.464012075 (433h4m42.33509511s) [seg_pos={0 0}, upload=380164]: Begin upload segment... Dec 23 16:33:42.464055629 (433h4m42.335138664s) [seg_pos={0 0}, upload=380164]: Done begin upload segment. Dec 23 16:33:42.464080826 (433h4m42.335163861s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464150278 (433h4m42.335233313s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464662895 (433h4m42.33574593s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464967405 (433h4m42.33605044s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465290650 (433h4m42.336373685s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465645145 (433h4m42.33672821s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465960345 (433h4m42.33704339s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466405773 (433h4m42.337488818s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466887240 (433h4m42.337970285s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467235243 (433h4m42.338318278s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467564109 (433h4m42.338647154s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467933462 (433h4m42.339016498s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.469791430 (433h4m42.340874465s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.470946911 (433h4m42.342029946s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.472385300 (433h4m42.343468335s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.473467610 (433h4m42.344550656s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:44.176460188 (433h4m44.047543253s) [upload=380164]: Next returned nil segment Dec 23 16:33:44.176479544 (433h4m44.047562599s) [upload=380164]: Waiting for error group managing segments... Dec 23 16:33:44.475711372 (433h4m44.346794407s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.477072674 (433h4m44.348155709s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.483581518 (433h4m44.354664553s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.489562778 (433h4m44.360645813s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.493900718 (433h4m44.364983753s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.504747682 (433h4m44.375830727s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.507440871 (433h4m44.378523916s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.726964291 (433h4m44.598047326s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.761338746 (433h4m44.632421781s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.807952966 (433h4m44.679036001s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.812606698 (433h4m44.683689733s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.965737354 (433h4m44.8368204s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373843518 (433h4m45.244926564s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373866111 (433h4m45.244949146s) [seg_pos={0 0}, upload=380164]: Segment reached optimal threshold of 13 pieces. Dec 23 16:33:45.374116078 (433h4m45.245199123s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374125225 (433h4m45.24520826s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374377926 (433h4m45.245460971s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374384488 (433h4m45.245467533s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374541508 (433h4m45.245624553s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374547108 (433h4m45.245630143s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374562777 (433h4m45.245645812s) [seg_pos={0 0}, upload=380164]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 16:33:45.374630947 (433h4m45.245713992s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 16:33:45.374637780 (433h4m45.245720815s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 16:33:45.672339675 (433h4m45.54342272s) [upload=380164]: Done uploading segments... Dec 23 16:33:45.672364471 (433h4m45.543447506s) [upload=380164]: Upload finished. err: <nil> Duration: 5.651309512s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........X.......................X.........X................] duration: 5.944176121s trace: https://jaeger.dev.storj.io/trace/7cce9d596d96c2fb FAIL (2025-12-23 16:46:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 23 16:33:42.217841516 (433h4m42.088924551s) [upload=380164]: Starting upload Dec 23 16:33:42.217925326 (433h4m42.089008361s) [upload=380164]: Done starting upload Dec 23 16:33:42.217975120 (433h4m42.089058155s) [upload=380164]: Uploading segments... Dec 23 16:33:42.218191021 (433h4m42.089274066s) [upload=380164]: Got next segment. Inline: false Dec 23 16:33:42.218228232 (433h4m42.089311267s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 16:33:42.218233072 (433h4m42.089316137s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 16:33:42.464012075 (433h4m42.33509511s) [seg_pos={0 0}, upload=380164]: Begin upload segment... Dec 23 16:33:42.464055629 (433h4m42.335138664s) [seg_pos={0 0}, upload=380164]: Done begin upload segment. Dec 23 16:33:42.464080826 (433h4m42.335163861s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464150278 (433h4m42.335233313s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464662895 (433h4m42.33574593s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464967405 (433h4m42.33605044s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465290650 (433h4m42.336373685s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465645145 (433h4m42.33672821s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465960345 (433h4m42.33704339s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466405773 (433h4m42.337488818s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466887240 (433h4m42.337970285s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467235243 (433h4m42.338318278s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467564109 (433h4m42.338647154s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467933462 (433h4m42.339016498s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.469791430 (433h4m42.340874465s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.470946911 (433h4m42.342029946s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.472385300 (433h4m42.343468335s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.473467610 (433h4m42.344550656s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:44.176460188 (433h4m44.047543253s) [upload=380164]: Next returned nil segment Dec 23 16:33:44.176479544 (433h4m44.047562599s) [upload=380164]: Waiting for error group managing segments... Dec 23 16:33:44.475711372 (433h4m44.346794407s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.477072674 (433h4m44.348155709s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.483581518 (433h4m44.354664553s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.489562778 (433h4m44.360645813s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.493900718 (433h4m44.364983753s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.504747682 (433h4m44.375830727s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.507440871 (433h4m44.378523916s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.726964291 (433h4m44.598047326s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.761338746 (433h4m44.632421781s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.807952966 (433h4m44.679036001s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.812606698 (433h4m44.683689733s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.965737354 (433h4m44.8368204s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373843518 (433h4m45.244926564s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373866111 (433h4m45.244949146s) [seg_pos={0 0}, upload=380164]: Segment reached optimal threshold of 13 pieces. Dec 23 16:33:45.374116078 (433h4m45.245199123s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374125225 (433h4m45.24520826s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374377926 (433h4m45.245460971s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374384488 (433h4m45.245467533s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374541508 (433h4m45.245624553s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374547108 (433h4m45.245630143s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374562777 (433h4m45.245645812s) [seg_pos={0 0}, upload=380164]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 16:33:45.374630947 (433h4m45.245713992s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 16:33:45.374637780 (433h4m45.245720815s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 16:33:45.672339675 (433h4m45.54342272s) [upload=380164]: Done uploading segments... Dec 23 16:33:45.672364471 (433h4m45.543447506s) [upload=380164]: Upload finished. err: <nil> Duration: 5.944201918s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..........X.......................X.........X...............] duration: 5.74445174s trace: https://jaeger.dev.storj.io/trace/7cce9d596d96c2fb FAIL (2025-12-23 16:46:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 23 16:33:42.217841516 (433h4m42.088924551s) [upload=380164]: Starting upload Dec 23 16:33:42.217925326 (433h4m42.089008361s) [upload=380164]: Done starting upload Dec 23 16:33:42.217975120 (433h4m42.089058155s) [upload=380164]: Uploading segments... Dec 23 16:33:42.218191021 (433h4m42.089274066s) [upload=380164]: Got next segment. Inline: false Dec 23 16:33:42.218228232 (433h4m42.089311267s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 16:33:42.218233072 (433h4m42.089316137s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 16:33:42.464012075 (433h4m42.33509511s) [seg_pos={0 0}, upload=380164]: Begin upload segment... Dec 23 16:33:42.464055629 (433h4m42.335138664s) [seg_pos={0 0}, upload=380164]: Done begin upload segment. Dec 23 16:33:42.464080826 (433h4m42.335163861s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464150278 (433h4m42.335233313s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464662895 (433h4m42.33574593s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464967405 (433h4m42.33605044s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465290650 (433h4m42.336373685s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465645145 (433h4m42.33672821s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465960345 (433h4m42.33704339s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466405773 (433h4m42.337488818s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466887240 (433h4m42.337970285s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467235243 (433h4m42.338318278s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467564109 (433h4m42.338647154s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467933462 (433h4m42.339016498s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.469791430 (433h4m42.340874465s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.470946911 (433h4m42.342029946s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.472385300 (433h4m42.343468335s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.473467610 (433h4m42.344550656s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:44.176460188 (433h4m44.047543253s) [upload=380164]: Next returned nil segment Dec 23 16:33:44.176479544 (433h4m44.047562599s) [upload=380164]: Waiting for error group managing segments... Dec 23 16:33:44.475711372 (433h4m44.346794407s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.477072674 (433h4m44.348155709s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.483581518 (433h4m44.354664553s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.489562778 (433h4m44.360645813s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.493900718 (433h4m44.364983753s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.504747682 (433h4m44.375830727s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.507440871 (433h4m44.378523916s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.726964291 (433h4m44.598047326s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.761338746 (433h4m44.632421781s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.807952966 (433h4m44.679036001s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.812606698 (433h4m44.683689733s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.965737354 (433h4m44.8368204s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373843518 (433h4m45.244926564s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373866111 (433h4m45.244949146s) [seg_pos={0 0}, upload=380164]: Segment reached optimal threshold of 13 pieces. Dec 23 16:33:45.374116078 (433h4m45.245199123s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374125225 (433h4m45.24520826s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374377926 (433h4m45.245460971s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374384488 (433h4m45.245467533s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374541508 (433h4m45.245624553s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374547108 (433h4m45.245630143s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374562777 (433h4m45.245645812s) [seg_pos={0 0}, upload=380164]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 16:33:45.374630947 (433h4m45.245713992s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 16:33:45.374637780 (433h4m45.245720815s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 16:33:45.672339675 (433h4m45.54342272s) [upload=380164]: Done uploading segments... Dec 23 16:33:45.672364471 (433h4m45.543447506s) [upload=380164]: Upload finished. err: <nil> Duration: 5.74447702s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...........X.......................X.........X..............] duration: 5.726964793s trace: https://jaeger.dev.storj.io/trace/7cce9d596d96c2fb FAIL (2025-12-23 16:45:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 23 16:33:42.217841516 (433h4m42.088924551s) [upload=380164]: Starting upload Dec 23 16:33:42.217925326 (433h4m42.089008361s) [upload=380164]: Done starting upload Dec 23 16:33:42.217975120 (433h4m42.089058155s) [upload=380164]: Uploading segments... Dec 23 16:33:42.218191021 (433h4m42.089274066s) [upload=380164]: Got next segment. Inline: false Dec 23 16:33:42.218228232 (433h4m42.089311267s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 16:33:42.218233072 (433h4m42.089316137s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 16:33:42.464012075 (433h4m42.33509511s) [seg_pos={0 0}, upload=380164]: Begin upload segment... Dec 23 16:33:42.464055629 (433h4m42.335138664s) [seg_pos={0 0}, upload=380164]: Done begin upload segment. Dec 23 16:33:42.464080826 (433h4m42.335163861s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464150278 (433h4m42.335233313s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464662895 (433h4m42.33574593s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464967405 (433h4m42.33605044s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465290650 (433h4m42.336373685s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465645145 (433h4m42.33672821s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465960345 (433h4m42.33704339s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466405773 (433h4m42.337488818s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466887240 (433h4m42.337970285s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467235243 (433h4m42.338318278s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467564109 (433h4m42.338647154s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467933462 (433h4m42.339016498s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.469791430 (433h4m42.340874465s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.470946911 (433h4m42.342029946s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.472385300 (433h4m42.343468335s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.473467610 (433h4m42.344550656s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:44.176460188 (433h4m44.047543253s) [upload=380164]: Next returned nil segment Dec 23 16:33:44.176479544 (433h4m44.047562599s) [upload=380164]: Waiting for error group managing segments... Dec 23 16:33:44.475711372 (433h4m44.346794407s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.477072674 (433h4m44.348155709s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.483581518 (433h4m44.354664553s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.489562778 (433h4m44.360645813s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.493900718 (433h4m44.364983753s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.504747682 (433h4m44.375830727s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.507440871 (433h4m44.378523916s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.726964291 (433h4m44.598047326s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.761338746 (433h4m44.632421781s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.807952966 (433h4m44.679036001s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.812606698 (433h4m44.683689733s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.965737354 (433h4m44.8368204s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373843518 (433h4m45.244926564s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373866111 (433h4m45.244949146s) [seg_pos={0 0}, upload=380164]: Segment reached optimal threshold of 13 pieces. Dec 23 16:33:45.374116078 (433h4m45.245199123s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374125225 (433h4m45.24520826s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374377926 (433h4m45.245460971s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374384488 (433h4m45.245467533s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374541508 (433h4m45.245624553s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374547108 (433h4m45.245630143s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374562777 (433h4m45.245645812s) [seg_pos={0 0}, upload=380164]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 16:33:45.374630947 (433h4m45.245713992s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 16:33:45.374637780 (433h4m45.245720815s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 16:33:45.672339675 (433h4m45.54342272s) [upload=380164]: Done uploading segments... Dec 23 16:33:45.672364471 (433h4m45.543447506s) [upload=380164]: Upload finished. err: <nil> Duration: 5.72699015s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [............X.......................X.........X.............] duration: 5.850940433s trace: https://jaeger.dev.storj.io/trace/7cce9d596d96c2fb FAIL (2025-12-23 16:45:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 23 16:33:42.217841516 (433h4m42.088924551s) [upload=380164]: Starting upload Dec 23 16:33:42.217925326 (433h4m42.089008361s) [upload=380164]: Done starting upload Dec 23 16:33:42.217975120 (433h4m42.089058155s) [upload=380164]: Uploading segments... Dec 23 16:33:42.218191021 (433h4m42.089274066s) [upload=380164]: Got next segment. Inline: false Dec 23 16:33:42.218228232 (433h4m42.089311267s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 16:33:42.218233072 (433h4m42.089316137s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 16:33:42.464012075 (433h4m42.33509511s) [seg_pos={0 0}, upload=380164]: Begin upload segment... Dec 23 16:33:42.464055629 (433h4m42.335138664s) [seg_pos={0 0}, upload=380164]: Done begin upload segment. Dec 23 16:33:42.464080826 (433h4m42.335163861s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464150278 (433h4m42.335233313s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464662895 (433h4m42.33574593s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464967405 (433h4m42.33605044s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465290650 (433h4m42.336373685s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465645145 (433h4m42.33672821s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465960345 (433h4m42.33704339s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466405773 (433h4m42.337488818s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466887240 (433h4m42.337970285s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467235243 (433h4m42.338318278s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467564109 (433h4m42.338647154s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467933462 (433h4m42.339016498s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.469791430 (433h4m42.340874465s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.470946911 (433h4m42.342029946s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.472385300 (433h4m42.343468335s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.473467610 (433h4m42.344550656s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:44.176460188 (433h4m44.047543253s) [upload=380164]: Next returned nil segment Dec 23 16:33:44.176479544 (433h4m44.047562599s) [upload=380164]: Waiting for error group managing segments... Dec 23 16:33:44.475711372 (433h4m44.346794407s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.477072674 (433h4m44.348155709s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.483581518 (433h4m44.354664553s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.489562778 (433h4m44.360645813s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.493900718 (433h4m44.364983753s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.504747682 (433h4m44.375830727s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.507440871 (433h4m44.378523916s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.726964291 (433h4m44.598047326s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.761338746 (433h4m44.632421781s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.807952966 (433h4m44.679036001s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.812606698 (433h4m44.683689733s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.965737354 (433h4m44.8368204s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373843518 (433h4m45.244926564s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373866111 (433h4m45.244949146s) [seg_pos={0 0}, upload=380164]: Segment reached optimal threshold of 13 pieces. Dec 23 16:33:45.374116078 (433h4m45.245199123s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374125225 (433h4m45.24520826s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374377926 (433h4m45.245460971s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374384488 (433h4m45.245467533s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374541508 (433h4m45.245624553s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374547108 (433h4m45.245630143s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374562777 (433h4m45.245645812s) [seg_pos={0 0}, upload=380164]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 16:33:45.374630947 (433h4m45.245713992s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 16:33:45.374637780 (433h4m45.245720815s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 16:33:45.672339675 (433h4m45.54342272s) [upload=380164]: Done uploading segments... Dec 23 16:33:45.672364471 (433h4m45.543447506s) [upload=380164]: Upload finished. err: <nil> Duration: 5.850971121s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.............X.......................X.........X............] duration: 5.607578603s trace: https://jaeger.dev.storj.io/trace/7cce9d596d96c2fb FAIL (2025-12-23 16:45:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 23 16:33:42.217841516 (433h4m42.088924551s) [upload=380164]: Starting upload Dec 23 16:33:42.217925326 (433h4m42.089008361s) [upload=380164]: Done starting upload Dec 23 16:33:42.217975120 (433h4m42.089058155s) [upload=380164]: Uploading segments... Dec 23 16:33:42.218191021 (433h4m42.089274066s) [upload=380164]: Got next segment. Inline: false Dec 23 16:33:42.218228232 (433h4m42.089311267s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 16:33:42.218233072 (433h4m42.089316137s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 16:33:42.464012075 (433h4m42.33509511s) [seg_pos={0 0}, upload=380164]: Begin upload segment... Dec 23 16:33:42.464055629 (433h4m42.335138664s) [seg_pos={0 0}, upload=380164]: Done begin upload segment. Dec 23 16:33:42.464080826 (433h4m42.335163861s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464150278 (433h4m42.335233313s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464662895 (433h4m42.33574593s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464967405 (433h4m42.33605044s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465290650 (433h4m42.336373685s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465645145 (433h4m42.33672821s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465960345 (433h4m42.33704339s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466405773 (433h4m42.337488818s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466887240 (433h4m42.337970285s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467235243 (433h4m42.338318278s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467564109 (433h4m42.338647154s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467933462 (433h4m42.339016498s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.469791430 (433h4m42.340874465s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.470946911 (433h4m42.342029946s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.472385300 (433h4m42.343468335s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.473467610 (433h4m42.344550656s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:44.176460188 (433h4m44.047543253s) [upload=380164]: Next returned nil segment Dec 23 16:33:44.176479544 (433h4m44.047562599s) [upload=380164]: Waiting for error group managing segments... Dec 23 16:33:44.475711372 (433h4m44.346794407s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.477072674 (433h4m44.348155709s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.483581518 (433h4m44.354664553s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.489562778 (433h4m44.360645813s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.493900718 (433h4m44.364983753s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.504747682 (433h4m44.375830727s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.507440871 (433h4m44.378523916s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.726964291 (433h4m44.598047326s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.761338746 (433h4m44.632421781s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.807952966 (433h4m44.679036001s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.812606698 (433h4m44.683689733s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.965737354 (433h4m44.8368204s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373843518 (433h4m45.244926564s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373866111 (433h4m45.244949146s) [seg_pos={0 0}, upload=380164]: Segment reached optimal threshold of 13 pieces. Dec 23 16:33:45.374116078 (433h4m45.245199123s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374125225 (433h4m45.24520826s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374377926 (433h4m45.245460971s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374384488 (433h4m45.245467533s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374541508 (433h4m45.245624553s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374547108 (433h4m45.245630143s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374562777 (433h4m45.245645812s) [seg_pos={0 0}, upload=380164]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 16:33:45.374630947 (433h4m45.245713992s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 16:33:45.374637780 (433h4m45.245720815s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 16:33:45.672339675 (433h4m45.54342272s) [upload=380164]: Done uploading segments... Dec 23 16:33:45.672364471 (433h4m45.543447506s) [upload=380164]: Upload finished. err: <nil> Duration: 5.607606806s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..............X.......................X.........X...........] duration: 5.69156162s trace: https://jaeger.dev.storj.io/trace/7cce9d596d96c2fb FAIL (2025-12-23 16:43:59 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 23 16:33:42.217841516 (433h4m42.088924551s) [upload=380164]: Starting upload Dec 23 16:33:42.217925326 (433h4m42.089008361s) [upload=380164]: Done starting upload Dec 23 16:33:42.217975120 (433h4m42.089058155s) [upload=380164]: Uploading segments... Dec 23 16:33:42.218191021 (433h4m42.089274066s) [upload=380164]: Got next segment. Inline: false Dec 23 16:33:42.218228232 (433h4m42.089311267s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 16:33:42.218233072 (433h4m42.089316137s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 16:33:42.464012075 (433h4m42.33509511s) [seg_pos={0 0}, upload=380164]: Begin upload segment... Dec 23 16:33:42.464055629 (433h4m42.335138664s) [seg_pos={0 0}, upload=380164]: Done begin upload segment. Dec 23 16:33:42.464080826 (433h4m42.335163861s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464150278 (433h4m42.335233313s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464662895 (433h4m42.33574593s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464967405 (433h4m42.33605044s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465290650 (433h4m42.336373685s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465645145 (433h4m42.33672821s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465960345 (433h4m42.33704339s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466405773 (433h4m42.337488818s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466887240 (433h4m42.337970285s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467235243 (433h4m42.338318278s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467564109 (433h4m42.338647154s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467933462 (433h4m42.339016498s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.469791430 (433h4m42.340874465s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.470946911 (433h4m42.342029946s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.472385300 (433h4m42.343468335s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.473467610 (433h4m42.344550656s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:44.176460188 (433h4m44.047543253s) [upload=380164]: Next returned nil segment Dec 23 16:33:44.176479544 (433h4m44.047562599s) [upload=380164]: Waiting for error group managing segments... Dec 23 16:33:44.475711372 (433h4m44.346794407s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.477072674 (433h4m44.348155709s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.483581518 (433h4m44.354664553s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.489562778 (433h4m44.360645813s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.493900718 (433h4m44.364983753s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.504747682 (433h4m44.375830727s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.507440871 (433h4m44.378523916s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.726964291 (433h4m44.598047326s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.761338746 (433h4m44.632421781s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.807952966 (433h4m44.679036001s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.812606698 (433h4m44.683689733s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.965737354 (433h4m44.8368204s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373843518 (433h4m45.244926564s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373866111 (433h4m45.244949146s) [seg_pos={0 0}, upload=380164]: Segment reached optimal threshold of 13 pieces. Dec 23 16:33:45.374116078 (433h4m45.245199123s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374125225 (433h4m45.24520826s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374377926 (433h4m45.245460971s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374384488 (433h4m45.245467533s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374541508 (433h4m45.245624553s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374547108 (433h4m45.245630143s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374562777 (433h4m45.245645812s) [seg_pos={0 0}, upload=380164]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 16:33:45.374630947 (433h4m45.245713992s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 16:33:45.374637780 (433h4m45.245720815s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 16:33:45.672339675 (433h4m45.54342272s) [upload=380164]: Done uploading segments... Dec 23 16:33:45.672364471 (433h4m45.543447506s) [upload=380164]: Upload finished. err: <nil> Duration: 5.691590615s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...............X.......................X.........X..........] duration: 5.705246665s trace: https://jaeger.dev.storj.io/trace/7cce9d596d96c2fb FAIL (2025-12-23 16:42:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 23 16:33:42.217841516 (433h4m42.088924551s) [upload=380164]: Starting upload Dec 23 16:33:42.217925326 (433h4m42.089008361s) [upload=380164]: Done starting upload Dec 23 16:33:42.217975120 (433h4m42.089058155s) [upload=380164]: Uploading segments... Dec 23 16:33:42.218191021 (433h4m42.089274066s) [upload=380164]: Got next segment. Inline: false Dec 23 16:33:42.218228232 (433h4m42.089311267s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 16:33:42.218233072 (433h4m42.089316137s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 16:33:42.464012075 (433h4m42.33509511s) [seg_pos={0 0}, upload=380164]: Begin upload segment... Dec 23 16:33:42.464055629 (433h4m42.335138664s) [seg_pos={0 0}, upload=380164]: Done begin upload segment. Dec 23 16:33:42.464080826 (433h4m42.335163861s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464150278 (433h4m42.335233313s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464662895 (433h4m42.33574593s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464967405 (433h4m42.33605044s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465290650 (433h4m42.336373685s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465645145 (433h4m42.33672821s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465960345 (433h4m42.33704339s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466405773 (433h4m42.337488818s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466887240 (433h4m42.337970285s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467235243 (433h4m42.338318278s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467564109 (433h4m42.338647154s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467933462 (433h4m42.339016498s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.469791430 (433h4m42.340874465s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.470946911 (433h4m42.342029946s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.472385300 (433h4m42.343468335s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.473467610 (433h4m42.344550656s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:44.176460188 (433h4m44.047543253s) [upload=380164]: Next returned nil segment Dec 23 16:33:44.176479544 (433h4m44.047562599s) [upload=380164]: Waiting for error group managing segments... Dec 23 16:33:44.475711372 (433h4m44.346794407s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.477072674 (433h4m44.348155709s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.483581518 (433h4m44.354664553s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.489562778 (433h4m44.360645813s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.493900718 (433h4m44.364983753s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.504747682 (433h4m44.375830727s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.507440871 (433h4m44.378523916s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.726964291 (433h4m44.598047326s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.761338746 (433h4m44.632421781s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.807952966 (433h4m44.679036001s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.812606698 (433h4m44.683689733s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.965737354 (433h4m44.8368204s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373843518 (433h4m45.244926564s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373866111 (433h4m45.244949146s) [seg_pos={0 0}, upload=380164]: Segment reached optimal threshold of 13 pieces. Dec 23 16:33:45.374116078 (433h4m45.245199123s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374125225 (433h4m45.24520826s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374377926 (433h4m45.245460971s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374384488 (433h4m45.245467533s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374541508 (433h4m45.245624553s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374547108 (433h4m45.245630143s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374562777 (433h4m45.245645812s) [seg_pos={0 0}, upload=380164]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 16:33:45.374630947 (433h4m45.245713992s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 16:33:45.374637780 (433h4m45.245720815s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 16:33:45.672339675 (433h4m45.54342272s) [upload=380164]: Done uploading segments... Dec 23 16:33:45.672364471 (433h4m45.543447506s) [upload=380164]: Upload finished. err: <nil> Duration: 5.70527497s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [................X.......................X.........X.........] duration: 5.86384168s trace: https://jaeger.dev.storj.io/trace/7cce9d596d96c2fb FAIL (2025-12-23 16:41:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 23 16:33:42.217841516 (433h4m42.088924551s) [upload=380164]: Starting upload Dec 23 16:33:42.217925326 (433h4m42.089008361s) [upload=380164]: Done starting upload Dec 23 16:33:42.217975120 (433h4m42.089058155s) [upload=380164]: Uploading segments... Dec 23 16:33:42.218191021 (433h4m42.089274066s) [upload=380164]: Got next segment. Inline: false Dec 23 16:33:42.218228232 (433h4m42.089311267s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 16:33:42.218233072 (433h4m42.089316137s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 16:33:42.464012075 (433h4m42.33509511s) [seg_pos={0 0}, upload=380164]: Begin upload segment... Dec 23 16:33:42.464055629 (433h4m42.335138664s) [seg_pos={0 0}, upload=380164]: Done begin upload segment. Dec 23 16:33:42.464080826 (433h4m42.335163861s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464150278 (433h4m42.335233313s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464662895 (433h4m42.33574593s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464967405 (433h4m42.33605044s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465290650 (433h4m42.336373685s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465645145 (433h4m42.33672821s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465960345 (433h4m42.33704339s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466405773 (433h4m42.337488818s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466887240 (433h4m42.337970285s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467235243 (433h4m42.338318278s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467564109 (433h4m42.338647154s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467933462 (433h4m42.339016498s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.469791430 (433h4m42.340874465s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.470946911 (433h4m42.342029946s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.472385300 (433h4m42.343468335s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.473467610 (433h4m42.344550656s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:44.176460188 (433h4m44.047543253s) [upload=380164]: Next returned nil segment Dec 23 16:33:44.176479544 (433h4m44.047562599s) [upload=380164]: Waiting for error group managing segments... Dec 23 16:33:44.475711372 (433h4m44.346794407s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.477072674 (433h4m44.348155709s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.483581518 (433h4m44.354664553s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.489562778 (433h4m44.360645813s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.493900718 (433h4m44.364983753s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.504747682 (433h4m44.375830727s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.507440871 (433h4m44.378523916s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.726964291 (433h4m44.598047326s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.761338746 (433h4m44.632421781s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.807952966 (433h4m44.679036001s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.812606698 (433h4m44.683689733s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.965737354 (433h4m44.8368204s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373843518 (433h4m45.244926564s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373866111 (433h4m45.244949146s) [seg_pos={0 0}, upload=380164]: Segment reached optimal threshold of 13 pieces. Dec 23 16:33:45.374116078 (433h4m45.245199123s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374125225 (433h4m45.24520826s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374377926 (433h4m45.245460971s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374384488 (433h4m45.245467533s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374541508 (433h4m45.245624553s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374547108 (433h4m45.245630143s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374562777 (433h4m45.245645812s) [seg_pos={0 0}, upload=380164]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 16:33:45.374630947 (433h4m45.245713992s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 16:33:45.374637780 (433h4m45.245720815s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 16:33:45.672339675 (433h4m45.54342272s) [upload=380164]: Done uploading segments... Dec 23 16:33:45.672364471 (433h4m45.543447506s) [upload=380164]: Upload finished. err: <nil> Duration: 5.863864524s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.................X.......................X.........X........] duration: 28.34057365s trace: https://jaeger.dev.storj.io/trace/7cce9d596d96c2fb FAIL (2025-12-23 16:41:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 23 16:33:42.217841516 (433h4m42.088924551s) [upload=380164]: Starting upload Dec 23 16:33:42.217925326 (433h4m42.089008361s) [upload=380164]: Done starting upload Dec 23 16:33:42.217975120 (433h4m42.089058155s) [upload=380164]: Uploading segments... Dec 23 16:33:42.218191021 (433h4m42.089274066s) [upload=380164]: Got next segment. Inline: false Dec 23 16:33:42.218228232 (433h4m42.089311267s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 16:33:42.218233072 (433h4m42.089316137s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 16:33:42.464012075 (433h4m42.33509511s) [seg_pos={0 0}, upload=380164]: Begin upload segment... Dec 23 16:33:42.464055629 (433h4m42.335138664s) [seg_pos={0 0}, upload=380164]: Done begin upload segment. Dec 23 16:33:42.464080826 (433h4m42.335163861s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464150278 (433h4m42.335233313s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464662895 (433h4m42.33574593s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464967405 (433h4m42.33605044s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465290650 (433h4m42.336373685s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465645145 (433h4m42.33672821s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465960345 (433h4m42.33704339s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466405773 (433h4m42.337488818s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466887240 (433h4m42.337970285s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467235243 (433h4m42.338318278s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467564109 (433h4m42.338647154s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467933462 (433h4m42.339016498s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.469791430 (433h4m42.340874465s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.470946911 (433h4m42.342029946s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.472385300 (433h4m42.343468335s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.473467610 (433h4m42.344550656s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:44.176460188 (433h4m44.047543253s) [upload=380164]: Next returned nil segment Dec 23 16:33:44.176479544 (433h4m44.047562599s) [upload=380164]: Waiting for error group managing segments... Dec 23 16:33:44.475711372 (433h4m44.346794407s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.477072674 (433h4m44.348155709s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.483581518 (433h4m44.354664553s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.489562778 (433h4m44.360645813s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.493900718 (433h4m44.364983753s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.504747682 (433h4m44.375830727s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.507440871 (433h4m44.378523916s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.726964291 (433h4m44.598047326s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.761338746 (433h4m44.632421781s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.807952966 (433h4m44.679036001s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.812606698 (433h4m44.683689733s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.965737354 (433h4m44.8368204s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373843518 (433h4m45.244926564s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373866111 (433h4m45.244949146s) [seg_pos={0 0}, upload=380164]: Segment reached optimal threshold of 13 pieces. Dec 23 16:33:45.374116078 (433h4m45.245199123s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374125225 (433h4m45.24520826s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374377926 (433h4m45.245460971s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374384488 (433h4m45.245467533s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374541508 (433h4m45.245624553s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374547108 (433h4m45.245630143s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374562777 (433h4m45.245645812s) [seg_pos={0 0}, upload=380164]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 16:33:45.374630947 (433h4m45.245713992s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 16:33:45.374637780 (433h4m45.245720815s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 16:33:45.672339675 (433h4m45.54342272s) [upload=380164]: Done uploading segments... Dec 23 16:33:45.672364471 (433h4m45.543447506s) [upload=380164]: Upload finished. err: <nil> Duration: 28.340612766s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [..................X.......................X.........X.......] duration: 5.841693817s trace: https://jaeger.dev.storj.io/trace/7cce9d596d96c2fb FAIL (2025-12-23 16:39:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 23 16:33:42.217841516 (433h4m42.088924551s) [upload=380164]: Starting upload Dec 23 16:33:42.217925326 (433h4m42.089008361s) [upload=380164]: Done starting upload Dec 23 16:33:42.217975120 (433h4m42.089058155s) [upload=380164]: Uploading segments... Dec 23 16:33:42.218191021 (433h4m42.089274066s) [upload=380164]: Got next segment. Inline: false Dec 23 16:33:42.218228232 (433h4m42.089311267s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 16:33:42.218233072 (433h4m42.089316137s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 16:33:42.464012075 (433h4m42.33509511s) [seg_pos={0 0}, upload=380164]: Begin upload segment... Dec 23 16:33:42.464055629 (433h4m42.335138664s) [seg_pos={0 0}, upload=380164]: Done begin upload segment. Dec 23 16:33:42.464080826 (433h4m42.335163861s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464150278 (433h4m42.335233313s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464662895 (433h4m42.33574593s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464967405 (433h4m42.33605044s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465290650 (433h4m42.336373685s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465645145 (433h4m42.33672821s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465960345 (433h4m42.33704339s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466405773 (433h4m42.337488818s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466887240 (433h4m42.337970285s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467235243 (433h4m42.338318278s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467564109 (433h4m42.338647154s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467933462 (433h4m42.339016498s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.469791430 (433h4m42.340874465s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.470946911 (433h4m42.342029946s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.472385300 (433h4m42.343468335s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.473467610 (433h4m42.344550656s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:44.176460188 (433h4m44.047543253s) [upload=380164]: Next returned nil segment Dec 23 16:33:44.176479544 (433h4m44.047562599s) [upload=380164]: Waiting for error group managing segments... Dec 23 16:33:44.475711372 (433h4m44.346794407s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.477072674 (433h4m44.348155709s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.483581518 (433h4m44.354664553s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.489562778 (433h4m44.360645813s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.493900718 (433h4m44.364983753s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.504747682 (433h4m44.375830727s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.507440871 (433h4m44.378523916s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.726964291 (433h4m44.598047326s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.761338746 (433h4m44.632421781s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.807952966 (433h4m44.679036001s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.812606698 (433h4m44.683689733s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.965737354 (433h4m44.8368204s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373843518 (433h4m45.244926564s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373866111 (433h4m45.244949146s) [seg_pos={0 0}, upload=380164]: Segment reached optimal threshold of 13 pieces. Dec 23 16:33:45.374116078 (433h4m45.245199123s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374125225 (433h4m45.24520826s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374377926 (433h4m45.245460971s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374384488 (433h4m45.245467533s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374541508 (433h4m45.245624553s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374547108 (433h4m45.245630143s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374562777 (433h4m45.245645812s) [seg_pos={0 0}, upload=380164]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 16:33:45.374630947 (433h4m45.245713992s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 16:33:45.374637780 (433h4m45.245720815s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 16:33:45.672339675 (433h4m45.54342272s) [upload=380164]: Done uploading segments... Dec 23 16:33:45.672364471 (433h4m45.543447506s) [upload=380164]: Upload finished. err: <nil> Duration: 5.841718434s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [...................X.......................X.........X......] duration: 5.939287941s trace: https://jaeger.dev.storj.io/trace/7cce9d596d96c2fb FAIL (2025-12-23 16:38:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 23 16:33:42.217841516 (433h4m42.088924551s) [upload=380164]: Starting upload Dec 23 16:33:42.217925326 (433h4m42.089008361s) [upload=380164]: Done starting upload Dec 23 16:33:42.217975120 (433h4m42.089058155s) [upload=380164]: Uploading segments... Dec 23 16:33:42.218191021 (433h4m42.089274066s) [upload=380164]: Got next segment. Inline: false Dec 23 16:33:42.218228232 (433h4m42.089311267s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 16:33:42.218233072 (433h4m42.089316137s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 16:33:42.464012075 (433h4m42.33509511s) [seg_pos={0 0}, upload=380164]: Begin upload segment... Dec 23 16:33:42.464055629 (433h4m42.335138664s) [seg_pos={0 0}, upload=380164]: Done begin upload segment. Dec 23 16:33:42.464080826 (433h4m42.335163861s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464150278 (433h4m42.335233313s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464662895 (433h4m42.33574593s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464967405 (433h4m42.33605044s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465290650 (433h4m42.336373685s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465645145 (433h4m42.33672821s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465960345 (433h4m42.33704339s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466405773 (433h4m42.337488818s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466887240 (433h4m42.337970285s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467235243 (433h4m42.338318278s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467564109 (433h4m42.338647154s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467933462 (433h4m42.339016498s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.469791430 (433h4m42.340874465s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.470946911 (433h4m42.342029946s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.472385300 (433h4m42.343468335s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.473467610 (433h4m42.344550656s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:44.176460188 (433h4m44.047543253s) [upload=380164]: Next returned nil segment Dec 23 16:33:44.176479544 (433h4m44.047562599s) [upload=380164]: Waiting for error group managing segments... Dec 23 16:33:44.475711372 (433h4m44.346794407s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.477072674 (433h4m44.348155709s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.483581518 (433h4m44.354664553s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.489562778 (433h4m44.360645813s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.493900718 (433h4m44.364983753s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.504747682 (433h4m44.375830727s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.507440871 (433h4m44.378523916s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.726964291 (433h4m44.598047326s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.761338746 (433h4m44.632421781s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.807952966 (433h4m44.679036001s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.812606698 (433h4m44.683689733s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.965737354 (433h4m44.8368204s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373843518 (433h4m45.244926564s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373866111 (433h4m45.244949146s) [seg_pos={0 0}, upload=380164]: Segment reached optimal threshold of 13 pieces. Dec 23 16:33:45.374116078 (433h4m45.245199123s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374125225 (433h4m45.24520826s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374377926 (433h4m45.245460971s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374384488 (433h4m45.245467533s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374541508 (433h4m45.245624553s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374547108 (433h4m45.245630143s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374562777 (433h4m45.245645812s) [seg_pos={0 0}, upload=380164]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 16:33:45.374630947 (433h4m45.245713992s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 16:33:45.374637780 (433h4m45.245720815s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 16:33:45.672339675 (433h4m45.54342272s) [upload=380164]: Done uploading segments... Dec 23 16:33:45.672364471 (433h4m45.543447506s) [upload=380164]: Upload finished. err: <nil> Duration: 5.939318521s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [....................X.......................X.........X.....] duration: 5.687756856s trace: https://jaeger.dev.storj.io/trace/7cce9d596d96c2fb FAIL (2025-12-23 16:37:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 23 16:33:42.217841516 (433h4m42.088924551s) [upload=380164]: Starting upload Dec 23 16:33:42.217925326 (433h4m42.089008361s) [upload=380164]: Done starting upload Dec 23 16:33:42.217975120 (433h4m42.089058155s) [upload=380164]: Uploading segments... Dec 23 16:33:42.218191021 (433h4m42.089274066s) [upload=380164]: Got next segment. Inline: false Dec 23 16:33:42.218228232 (433h4m42.089311267s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 16:33:42.218233072 (433h4m42.089316137s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 16:33:42.464012075 (433h4m42.33509511s) [seg_pos={0 0}, upload=380164]: Begin upload segment... Dec 23 16:33:42.464055629 (433h4m42.335138664s) [seg_pos={0 0}, upload=380164]: Done begin upload segment. Dec 23 16:33:42.464080826 (433h4m42.335163861s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464150278 (433h4m42.335233313s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464662895 (433h4m42.33574593s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464967405 (433h4m42.33605044s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465290650 (433h4m42.336373685s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465645145 (433h4m42.33672821s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465960345 (433h4m42.33704339s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466405773 (433h4m42.337488818s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466887240 (433h4m42.337970285s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467235243 (433h4m42.338318278s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467564109 (433h4m42.338647154s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467933462 (433h4m42.339016498s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.469791430 (433h4m42.340874465s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.470946911 (433h4m42.342029946s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.472385300 (433h4m42.343468335s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.473467610 (433h4m42.344550656s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:44.176460188 (433h4m44.047543253s) [upload=380164]: Next returned nil segment Dec 23 16:33:44.176479544 (433h4m44.047562599s) [upload=380164]: Waiting for error group managing segments... Dec 23 16:33:44.475711372 (433h4m44.346794407s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.477072674 (433h4m44.348155709s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.483581518 (433h4m44.354664553s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.489562778 (433h4m44.360645813s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.493900718 (433h4m44.364983753s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.504747682 (433h4m44.375830727s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.507440871 (433h4m44.378523916s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.726964291 (433h4m44.598047326s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.761338746 (433h4m44.632421781s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.807952966 (433h4m44.679036001s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.812606698 (433h4m44.683689733s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.965737354 (433h4m44.8368204s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373843518 (433h4m45.244926564s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373866111 (433h4m45.244949146s) [seg_pos={0 0}, upload=380164]: Segment reached optimal threshold of 13 pieces. Dec 23 16:33:45.374116078 (433h4m45.245199123s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374125225 (433h4m45.24520826s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374377926 (433h4m45.245460971s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374384488 (433h4m45.245467533s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374541508 (433h4m45.245624553s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374547108 (433h4m45.245630143s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374562777 (433h4m45.245645812s) [seg_pos={0 0}, upload=380164]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 16:33:45.374630947 (433h4m45.245713992s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 16:33:45.374637780 (433h4m45.245720815s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 16:33:45.672339675 (433h4m45.54342272s) [upload=380164]: Done uploading segments... Dec 23 16:33:45.672364471 (433h4m45.543447506s) [upload=380164]: Upload finished. err: <nil> Duration: 5.687781331s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.....................X.......................X.........X....] duration: 5.75655038s trace: https://jaeger.dev.storj.io/trace/7cce9d596d96c2fb FAIL (2025-12-23 16:37:13 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 23 16:33:42.217841516 (433h4m42.088924551s) [upload=380164]: Starting upload Dec 23 16:33:42.217925326 (433h4m42.089008361s) [upload=380164]: Done starting upload Dec 23 16:33:42.217975120 (433h4m42.089058155s) [upload=380164]: Uploading segments... Dec 23 16:33:42.218191021 (433h4m42.089274066s) [upload=380164]: Got next segment. Inline: false Dec 23 16:33:42.218228232 (433h4m42.089311267s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 16:33:42.218233072 (433h4m42.089316137s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 16:33:42.464012075 (433h4m42.33509511s) [seg_pos={0 0}, upload=380164]: Begin upload segment... Dec 23 16:33:42.464055629 (433h4m42.335138664s) [seg_pos={0 0}, upload=380164]: Done begin upload segment. Dec 23 16:33:42.464080826 (433h4m42.335163861s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464150278 (433h4m42.335233313s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464662895 (433h4m42.33574593s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464967405 (433h4m42.33605044s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465290650 (433h4m42.336373685s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465645145 (433h4m42.33672821s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465960345 (433h4m42.33704339s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466405773 (433h4m42.337488818s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466887240 (433h4m42.337970285s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467235243 (433h4m42.338318278s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467564109 (433h4m42.338647154s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467933462 (433h4m42.339016498s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.469791430 (433h4m42.340874465s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.470946911 (433h4m42.342029946s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.472385300 (433h4m42.343468335s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.473467610 (433h4m42.344550656s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:44.176460188 (433h4m44.047543253s) [upload=380164]: Next returned nil segment Dec 23 16:33:44.176479544 (433h4m44.047562599s) [upload=380164]: Waiting for error group managing segments... Dec 23 16:33:44.475711372 (433h4m44.346794407s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.477072674 (433h4m44.348155709s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.483581518 (433h4m44.354664553s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.489562778 (433h4m44.360645813s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.493900718 (433h4m44.364983753s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.504747682 (433h4m44.375830727s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.507440871 (433h4m44.378523916s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.726964291 (433h4m44.598047326s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.761338746 (433h4m44.632421781s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.807952966 (433h4m44.679036001s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.812606698 (433h4m44.683689733s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.965737354 (433h4m44.8368204s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373843518 (433h4m45.244926564s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373866111 (433h4m45.244949146s) [seg_pos={0 0}, upload=380164]: Segment reached optimal threshold of 13 pieces. Dec 23 16:33:45.374116078 (433h4m45.245199123s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374125225 (433h4m45.24520826s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374377926 (433h4m45.245460971s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374384488 (433h4m45.245467533s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374541508 (433h4m45.245624553s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374547108 (433h4m45.245630143s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374562777 (433h4m45.245645812s) [seg_pos={0 0}, upload=380164]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 16:33:45.374630947 (433h4m45.245713992s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 16:33:45.374637780 (433h4m45.245720815s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 16:33:45.672339675 (433h4m45.54342272s) [upload=380164]: Done uploading segments... Dec 23 16:33:45.672364471 (433h4m45.543447506s) [upload=380164]: Upload finished. err: <nil> Duration: 5.75657676s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [......................X.......................X.........X...] duration: 5.692782544s trace: https://jaeger.dev.storj.io/trace/7cce9d596d96c2fb FAIL (2025-12-23 16:36:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 23 16:33:42.217841516 (433h4m42.088924551s) [upload=380164]: Starting upload Dec 23 16:33:42.217925326 (433h4m42.089008361s) [upload=380164]: Done starting upload Dec 23 16:33:42.217975120 (433h4m42.089058155s) [upload=380164]: Uploading segments... Dec 23 16:33:42.218191021 (433h4m42.089274066s) [upload=380164]: Got next segment. Inline: false Dec 23 16:33:42.218228232 (433h4m42.089311267s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 16:33:42.218233072 (433h4m42.089316137s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 16:33:42.464012075 (433h4m42.33509511s) [seg_pos={0 0}, upload=380164]: Begin upload segment... Dec 23 16:33:42.464055629 (433h4m42.335138664s) [seg_pos={0 0}, upload=380164]: Done begin upload segment. Dec 23 16:33:42.464080826 (433h4m42.335163861s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464150278 (433h4m42.335233313s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464662895 (433h4m42.33574593s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464967405 (433h4m42.33605044s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465290650 (433h4m42.336373685s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465645145 (433h4m42.33672821s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465960345 (433h4m42.33704339s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466405773 (433h4m42.337488818s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466887240 (433h4m42.337970285s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467235243 (433h4m42.338318278s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467564109 (433h4m42.338647154s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467933462 (433h4m42.339016498s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.469791430 (433h4m42.340874465s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.470946911 (433h4m42.342029946s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.472385300 (433h4m42.343468335s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.473467610 (433h4m42.344550656s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:44.176460188 (433h4m44.047543253s) [upload=380164]: Next returned nil segment Dec 23 16:33:44.176479544 (433h4m44.047562599s) [upload=380164]: Waiting for error group managing segments... Dec 23 16:33:44.475711372 (433h4m44.346794407s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.477072674 (433h4m44.348155709s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.483581518 (433h4m44.354664553s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.489562778 (433h4m44.360645813s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.493900718 (433h4m44.364983753s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.504747682 (433h4m44.375830727s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.507440871 (433h4m44.378523916s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.726964291 (433h4m44.598047326s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.761338746 (433h4m44.632421781s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.807952966 (433h4m44.679036001s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.812606698 (433h4m44.683689733s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.965737354 (433h4m44.8368204s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373843518 (433h4m45.244926564s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373866111 (433h4m45.244949146s) [seg_pos={0 0}, upload=380164]: Segment reached optimal threshold of 13 pieces. Dec 23 16:33:45.374116078 (433h4m45.245199123s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374125225 (433h4m45.24520826s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374377926 (433h4m45.245460971s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374384488 (433h4m45.245467533s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374541508 (433h4m45.245624553s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374547108 (433h4m45.245630143s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374562777 (433h4m45.245645812s) [seg_pos={0 0}, upload=380164]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 16:33:45.374630947 (433h4m45.245713992s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 16:33:45.374637780 (433h4m45.245720815s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 16:33:45.672339675 (433h4m45.54342272s) [upload=380164]: Done uploading segments... Dec 23 16:33:45.672364471 (433h4m45.543447506s) [upload=380164]: Upload finished. err: <nil> Duration: 5.692811969s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.......................X.......................X.........X..] duration: 6.406322635s trace: https://jaeger.dev.storj.io/trace/7cce9d596d96c2fb FAIL (2025-12-23 16:35:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 23 16:33:42.217841516 (433h4m42.088924551s) [upload=380164]: Starting upload Dec 23 16:33:42.217925326 (433h4m42.089008361s) [upload=380164]: Done starting upload Dec 23 16:33:42.217975120 (433h4m42.089058155s) [upload=380164]: Uploading segments... Dec 23 16:33:42.218191021 (433h4m42.089274066s) [upload=380164]: Got next segment. Inline: false Dec 23 16:33:42.218228232 (433h4m42.089311267s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 16:33:42.218233072 (433h4m42.089316137s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 16:33:42.464012075 (433h4m42.33509511s) [seg_pos={0 0}, upload=380164]: Begin upload segment... Dec 23 16:33:42.464055629 (433h4m42.335138664s) [seg_pos={0 0}, upload=380164]: Done begin upload segment. Dec 23 16:33:42.464080826 (433h4m42.335163861s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464150278 (433h4m42.335233313s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464662895 (433h4m42.33574593s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464967405 (433h4m42.33605044s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465290650 (433h4m42.336373685s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465645145 (433h4m42.33672821s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465960345 (433h4m42.33704339s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466405773 (433h4m42.337488818s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466887240 (433h4m42.337970285s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467235243 (433h4m42.338318278s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467564109 (433h4m42.338647154s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467933462 (433h4m42.339016498s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.469791430 (433h4m42.340874465s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.470946911 (433h4m42.342029946s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.472385300 (433h4m42.343468335s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.473467610 (433h4m42.344550656s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:44.176460188 (433h4m44.047543253s) [upload=380164]: Next returned nil segment Dec 23 16:33:44.176479544 (433h4m44.047562599s) [upload=380164]: Waiting for error group managing segments... Dec 23 16:33:44.475711372 (433h4m44.346794407s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.477072674 (433h4m44.348155709s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.483581518 (433h4m44.354664553s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.489562778 (433h4m44.360645813s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.493900718 (433h4m44.364983753s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.504747682 (433h4m44.375830727s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.507440871 (433h4m44.378523916s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.726964291 (433h4m44.598047326s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.761338746 (433h4m44.632421781s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.807952966 (433h4m44.679036001s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.812606698 (433h4m44.683689733s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.965737354 (433h4m44.8368204s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373843518 (433h4m45.244926564s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373866111 (433h4m45.244949146s) [seg_pos={0 0}, upload=380164]: Segment reached optimal threshold of 13 pieces. Dec 23 16:33:45.374116078 (433h4m45.245199123s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374125225 (433h4m45.24520826s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374377926 (433h4m45.245460971s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374384488 (433h4m45.245467533s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374541508 (433h4m45.245624553s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374547108 (433h4m45.245630143s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374562777 (433h4m45.245645812s) [seg_pos={0 0}, upload=380164]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 16:33:45.374630947 (433h4m45.245713992s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 16:33:45.374637780 (433h4m45.245720815s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 16:33:45.672339675 (433h4m45.54342272s) [upload=380164]: Done uploading segments... Dec 23 16:33:45.672364471 (433h4m45.543447506s) [upload=380164]: Upload finished. err: <nil> Duration: 6.406354347s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [........................X.......................X.........X.] duration: 5.697754693s trace: https://jaeger.dev.storj.io/trace/7cce9d596d96c2fb FAIL (2025-12-23 16:34:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 23 16:33:42.217841516 (433h4m42.088924551s) [upload=380164]: Starting upload Dec 23 16:33:42.217925326 (433h4m42.089008361s) [upload=380164]: Done starting upload Dec 23 16:33:42.217975120 (433h4m42.089058155s) [upload=380164]: Uploading segments... Dec 23 16:33:42.218191021 (433h4m42.089274066s) [upload=380164]: Got next segment. Inline: false Dec 23 16:33:42.218228232 (433h4m42.089311267s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 16:33:42.218233072 (433h4m42.089316137s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 16:33:42.464012075 (433h4m42.33509511s) [seg_pos={0 0}, upload=380164]: Begin upload segment... Dec 23 16:33:42.464055629 (433h4m42.335138664s) [seg_pos={0 0}, upload=380164]: Done begin upload segment. Dec 23 16:33:42.464080826 (433h4m42.335163861s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464150278 (433h4m42.335233313s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464662895 (433h4m42.33574593s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464967405 (433h4m42.33605044s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465290650 (433h4m42.336373685s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465645145 (433h4m42.33672821s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465960345 (433h4m42.33704339s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466405773 (433h4m42.337488818s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466887240 (433h4m42.337970285s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467235243 (433h4m42.338318278s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467564109 (433h4m42.338647154s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467933462 (433h4m42.339016498s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.469791430 (433h4m42.340874465s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.470946911 (433h4m42.342029946s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.472385300 (433h4m42.343468335s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.473467610 (433h4m42.344550656s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:44.176460188 (433h4m44.047543253s) [upload=380164]: Next returned nil segment Dec 23 16:33:44.176479544 (433h4m44.047562599s) [upload=380164]: Waiting for error group managing segments... Dec 23 16:33:44.475711372 (433h4m44.346794407s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.477072674 (433h4m44.348155709s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.483581518 (433h4m44.354664553s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.489562778 (433h4m44.360645813s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.493900718 (433h4m44.364983753s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.504747682 (433h4m44.375830727s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.507440871 (433h4m44.378523916s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.726964291 (433h4m44.598047326s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.761338746 (433h4m44.632421781s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.807952966 (433h4m44.679036001s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.812606698 (433h4m44.683689733s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.965737354 (433h4m44.8368204s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373843518 (433h4m45.244926564s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373866111 (433h4m45.244949146s) [seg_pos={0 0}, upload=380164]: Segment reached optimal threshold of 13 pieces. Dec 23 16:33:45.374116078 (433h4m45.245199123s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374125225 (433h4m45.24520826s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374377926 (433h4m45.245460971s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374384488 (433h4m45.245467533s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374541508 (433h4m45.245624553s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374547108 (433h4m45.245630143s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374562777 (433h4m45.245645812s) [seg_pos={0 0}, upload=380164]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 16:33:45.374630947 (433h4m45.245713992s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 16:33:45.374637780 (433h4m45.245720815s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 16:33:45.672339675 (433h4m45.54342272s) [upload=380164]: Done uploading segments... Dec 23 16:33:45.672364471 (433h4m45.543447506s) [upload=380164]: Upload finished. err: <nil> Duration: 5.6977954s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.........................X.......................X.........X] duration: 30.003294916s trace: https://jaeger.dev.storj.io/trace/7cce9d596d96c2fb FAIL (2025-12-23 16:34:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 23 16:33:42.217841516 (433h4m42.088924551s) [upload=380164]: Starting upload Dec 23 16:33:42.217925326 (433h4m42.089008361s) [upload=380164]: Done starting upload Dec 23 16:33:42.217975120 (433h4m42.089058155s) [upload=380164]: Uploading segments... Dec 23 16:33:42.218191021 (433h4m42.089274066s) [upload=380164]: Got next segment. Inline: false Dec 23 16:33:42.218228232 (433h4m42.089311267s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 16:33:42.218233072 (433h4m42.089316137s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 16:33:42.464012075 (433h4m42.33509511s) [seg_pos={0 0}, upload=380164]: Begin upload segment... Dec 23 16:33:42.464055629 (433h4m42.335138664s) [seg_pos={0 0}, upload=380164]: Done begin upload segment. Dec 23 16:33:42.464080826 (433h4m42.335163861s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464150278 (433h4m42.335233313s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464662895 (433h4m42.33574593s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.464967405 (433h4m42.33605044s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465290650 (433h4m42.336373685s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465645145 (433h4m42.33672821s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.465960345 (433h4m42.33704339s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466405773 (433h4m42.337488818s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.466887240 (433h4m42.337970285s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467235243 (433h4m42.338318278s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467564109 (433h4m42.338647154s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.467933462 (433h4m42.339016498s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.469791430 (433h4m42.340874465s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.470946911 (433h4m42.342029946s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.472385300 (433h4m42.343468335s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:42.473467610 (433h4m42.344550656s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Uploading piece... Dec 23 16:33:44.176460188 (433h4m44.047543253s) [upload=380164]: Next returned nil segment Dec 23 16:33:44.176479544 (433h4m44.047562599s) [upload=380164]: Waiting for error group managing segments... Dec 23 16:33:44.475711372 (433h4m44.346794407s) [noise=true , address=145.79.227.51:4409 , piece_id=5YRQENSBNQKDYKKQ5CDFIFQPNGDDVWN7OCQY5VQNQV46HWGD3IBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.477072674 (433h4m44.348155709s) [noise=true , address=145.79.227.50:4373 , piece_id=LY4RCZKISHOS3PTD6A7RVOGKWH5SCSWVYJTOYKGBILMUPUV5ZU2Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.483581518 (433h4m44.354664553s) [noise=true , address=145.79.227.29:4942 , piece_id=AZCEHKSMVTC6OTRZCBGUXZPPASET3FZOIEE36FRXRGOYTXQG7G6Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.489562778 (433h4m44.360645813s) [noise=true , address=145.79.227.45:4229 , piece_id=J7YNBTHU6VPL2MWPX6MY7TPZUG2ITITEVAPPVIXYLYAJLA6JYCFQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.493900718 (433h4m44.364983753s) [noise=true , address=145.79.227.9:4367 , piece_id=PFU2G3QCT3TSRIUCIRDS2JMFX24KJXLCW3AIABIWY2GH4MNE6EUA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.504747682 (433h4m44.375830727s) [noise=true , address=145.79.227.14:4838 , piece_id=GLCTIO7S7DCMSKLJXYVMEFYI5E4KIQKIENEGPCGANXKSOAVIAVFA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.507440871 (433h4m44.378523916s) [noise=true , address=145.79.227.39:4014 , piece_id=L5TNIGODMLHAKHEFHUAY25XOGHRKF2TJDWEU523V5NV4JMF7VJVQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.726964291 (433h4m44.598047326s) [noise=true , address=45.91.116.14:4192 , piece_id=AKTS34A6D6GH634DP2RFIPM4V5H7RWPTHJ52CMCYG4P3TGKJOXOQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.761338746 (433h4m44.632421781s) [noise=true , address=45.91.116.9:4261 , piece_id=V5CVCFH4TTBZTRLVHATFQLOMS5JOASENETIKY6SYENF5NVYODP7Q, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.807952966 (433h4m44.679036001s) [noise=true , address=45.91.116.4:4940 , piece_id=PCESROSFQ6SI3IQGYJXW5IN2RSPVH7BJXPJBDAYWU72FKL6M3NGQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.812606698 (433h4m44.683689733s) [noise=true , address=45.91.116.10:4084 , piece_id=H2GOTOC7H436T5BLBJTXFOWYHWKODXEZK66GNPTWCIOHYGSQARAQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:44.965737354 (433h4m44.8368204s) [noise=true , address=145.79.227.34:4906 , piece_id=26BRURF3WG3ZYOVCYZSB5DOMMZWOGYTIOFP4Q6CRD3RYHTUB56DQ, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373843518 (433h4m45.244926564s) [noise=true , address=45.91.116.12:4228 , piece_id=5U7FPANH6LYIWHQRLOP5KZCECWPTFRWSFS6NEWJL2FF6EN3KKDBA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: <nil> Dec 23 16:33:45.373866111 (433h4m45.244949146s) [seg_pos={0 0}, upload=380164]: Segment reached optimal threshold of 13 pieces. Dec 23 16:33:45.374116078 (433h4m45.245199123s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:14LxqUFh6udxGJH9qJV6d1tdGgxyzv3qiZYENLtXffBCpLeG9h): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374125225 (433h4m45.24520826s) [noise=true , address=77.90.31.16:4735 , piece_id=F5D7JRGATUSEUBGRKKQ5QYY7KAOQC2KZSA6I5FOMQDS7YPSHSFCA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374377926 (433h4m45.245460971s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374384488 (433h4m45.245467533s) [noise=true , address=77.90.31.12:4590 , piece_id=ZXKZ6PAFEVTKISGFFVMKASULQP4EYJTANF4ALUASGFN62UO4GNSA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374541508 (433h4m45.245624553s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW): protocol: expected piece hash; context canceled; EOF Dec 23 16:33:45.374547108 (433h4m45.245630143s) [noise=true , address=45.91.116.5:4297 , piece_id=C4WYY2P33O2FZQSAFKLJ7EIOBDURVATLBN32ZYIRJHTUVYNJR3IA, seg_pos={0 0}, upload=380164]: Long tail cancellation due to optimal threshold Dec 23 16:33:45.374562777 (433h4m45.245645812s) [seg_pos={0 0}, upload=380164]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 16:33:45.374630947 (433h4m45.245713992s) [upload=380164]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 16:33:45.374637780 (433h4m45.245720815s) [upload=380164]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 16:33:45.672339675 (433h4m45.54342272s) [upload=380164]: Done uploading segments... Dec 23 16:33:45.672364471 (433h4m45.543447506s) [upload=380164]: Upload finished. err: <nil> Duration: 30.003353168s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................X.......................X.........] duration: 5.76797833s trace: https://jaeger.dev.storj.io/trace/31d899d501e5b690 FAIL (2025-12-23 16:33:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12pba8P3R4hb1xBifPDXzMKXuWgwwpPCx4MXPuBDU7Nupj2LKLW, Piece ID: OTH2HGKEPNWRXPTUUNE53VP46TTZC3DHIGQO6XPKBYDJL4SX5SJA) context deadline exceeded; piecestore: (Node ID: 12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, Piece ID: U3LNJVAFEK6QXLPLLPXIRMOANHJGAAH7IVUU66QXUVSIZV4TL62A) context deadline exceeded ======LOGS====== 16:25:53.398784201 (432h56m53.269867246s) [upload=380056]: Starting upload Dec 23 16:25:53.398856158 (432h56m53.269939193s) [upload=380056]: Done starting upload Dec 23 16:25:53.398995363 (432h56m53.270078408s) [upload=380056]: Uploading segments... Dec 23 16:25:53.399151801 (432h56m53.270234846s) [upload=380056]: Got next segment. Inline: false Dec 23 16:25:53.399189383 (432h56m53.270272438s) [upload=380056]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 23 16:25:53.399197348 (432h56m53.270280393s) [upload=380056]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 23 16:25:53.642014742 (432h56m53.513097777s) [seg_pos={0 0}, upload=380056]: Begin upload segment... Dec 23 16:25:53.642062833 (432h56m53.513145868s) [seg_pos={0 0}, upload=380056]: Done begin upload segment. Dec 23 16:25:53.642093792 (432h56m53.513176827s) [noise=true , address=45.91.116.14:4191 , piece_id=MJLISQSYJFUGWWK46OTMUZCZ2VQQ4EHS3S7YBFKCD4T3YXMWJVVA, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.642173464 (432h56m53.513256499s) [noise=true , address=45.91.116.10:4084 , piece_id=HAAU43H7A3UTI5XGHBFF6CEWCKWZ57KW4YTUHZZXUIXTUAN45VZA, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.642780630 (432h56m53.513863665s) [noise=true , address=77.90.31.11:4554 , piece_id=MEALTULOBOHZOXE6IA3Q4HXU4UH2DXUOEI2PAROQGTYWT4KGMJVQ, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.643204447 (432h56m53.514287482s) [noise=true , address=145.79.227.52:4446 , piece_id=EKVNZXJUVSXL7K4OT5URQHS2HQEZUSDFYBNAMHDJPMEYOCENQQHQ, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.643552189 (432h56m53.514635234s) [noise=true , address=145.79.227.40:4050 , piece_id=SI34DQR2C756H4ITMA5Z3MXWS2ZJGCKH3W232BB2OBG3LUWDDGIA, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.643897837 (432h56m53.514980882s) [noise=true , address=77.90.31.21:4878 , piece_id=U3LNJVAFEK6QXLPLLPXIRMOANHJGAAH7IVUU66QXUVSIZV4TL62A, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.644251771 (432h56m53.515334816s) [noise=true , address=145.79.227.38:4979 , piece_id=LPK3J2H6D2WKKAZZND73MMU76FQQDKFXRWSKPC5UTU4ECXAVY5GA, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.644562633 (432h56m53.515645668s) [noise=true , address=145.79.227.2:4113 , piece_id=GLGOU2EKRWKTT4IFIDHFA6IPNXUHONDBKUZYXPM5F73CEB57SFCQ, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.644921095 (432h56m53.51600414s) [noise=true , address=45.91.116.7:4010 , piece_id=PM2Q63VC7EG53EOXLJSAJC5O6QCFVOAPZKYY6JZX5WM7PK7PTS6A, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.645245584 (432h56m53.516328619s) [noise=true , address=45.91.116.9:4261 , piece_id=EOTTVRXGA3I3CQYZV5IR6N6O4CUEQSQW6WJTCD63DYRFPDPY4BVA, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.645270281 (432h56m53.516353326s) [noise=true , address=145.79.227.7:4293 , piece_id=P7GALY6OAOGJE6HVQDK4HLKA4FWRDXHRQRCWR7OJQVF3DO2PNOHA, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.647932911 (432h56m53.519015956s) [noise=true , address=145.79.227.49:4337 , piece_id=BDOJ6II43XR3WYVDF4MV6GSE7Q3VUBJ452PRVLRAGOBBGUSSJTCQ, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.649339679 (432h56m53.520422734s) [noise=true , address=45.91.116.13:4153 , piece_id=OFFHXVZIRZXXZZMAI2QHKNE56YJHHXEYTTW5S74UFGZFPBCFY62A, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.650673500 (432h56m53.521756535s) [noise=true , address=145.79.227.37:4567 , piece_id=HL7RAOBXZPCJI4TBVW2RFBBQHHG6EVMBIXCNHBTC6JWOVVLUDI7A, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.655923806 (432h56m53.527006841s) [noise=true , address=145.79.227.50:4375 , piece_id=2WFVPIJTRQR5KLZHZ65GZEPQBRVZFFETGGJVAFAW6G5QZ3LMHQGQ, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:53.656491558 (432h56m53.527574593s) [noise=true , address=45.91.116.5:4297 , piece_id=OTH2HGKEPNWRXPTUUNE53VP46TTZC3DHIGQO6XPKBYDJL4SX5SJA, seg_pos={0 0}, upload=380056]: Uploading piece... Dec 23 16:25:55.369187478 (432h56m55.240270513s) [upload=380056]: Next returned nil segment Dec 23 16:25:55.369207726 (432h56m55.240290761s) [upload=380056]: Waiting for error group managing segments... Dec 23 16:25:55.657604691 (432h56m55.528687726s) [noise=true , address=145.79.227.49:4337 , piece_id=BDOJ6II43XR3WYVDF4MV6GSE7Q3VUBJ452PRVLRAGOBBGUSSJTCQ, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:55.657891607 (432h56m55.528974642s) [noise=true , address=145.79.227.2:4113 , piece_id=GLGOU2EKRWKTT4IFIDHFA6IPNXUHONDBKUZYXPM5F73CEB57SFCQ, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:55.659128643 (432h56m55.530211678s) [noise=true , address=145.79.227.7:4293 , piece_id=P7GALY6OAOGJE6HVQDK4HLKA4FWRDXHRQRCWR7OJQVF3DO2PNOHA, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:55.671323092 (432h56m55.542406127s) [noise=true , address=145.79.227.38:4979 , piece_id=LPK3J2H6D2WKKAZZND73MMU76FQQDKFXRWSKPC5UTU4ECXAVY5GA, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:55.685990522 (432h56m55.557073557s) [noise=true , address=145.79.227.50:4375 , piece_id=2WFVPIJTRQR5KLZHZ65GZEPQBRVZFFETGGJVAFAW6G5QZ3LMHQGQ, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:55.686327192 (432h56m55.557410227s) [noise=true , address=145.79.227.37:4567 , piece_id=HL7RAOBXZPCJI4TBVW2RFBBQHHG6EVMBIXCNHBTC6JWOVVLUDI7A, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:55.749222355 (432h56m55.62030539s) [noise=true , address=145.79.227.52:4446 , piece_id=EKVNZXJUVSXL7K4OT5URQHS2HQEZUSDFYBNAMHDJPMEYOCENQQHQ, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:55.885581064 (432h56m55.756664109s) [noise=true , address=45.91.116.5:4297 , piece_id=OTH2HGKEPNWRXPTUUNE53VP46TTZC3DHIGQO6XPKBYDJL4SX5SJA, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:55.956015122 (432h56m55.827098157s) [noise=true , address=45.91.116.13:4153 , piece_id=OFFHXVZIRZXXZZMAI2QHKNE56YJHHXEYTTW5S74UFGZFPBCFY62A, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:56.104770316 (432h56m55.975853351s) [noise=true , address=77.90.31.21:4878 , piece_id=U3LNJVAFEK6QXLPLLPXIRMOANHJGAAH7IVUU66QXUVSIZV4TL62A, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:56.109411924 (432h56m55.980494959s) [noise=true , address=45.91.116.7:4010 , piece_id=PM2Q63VC7EG53EOXLJSAJC5O6QCFVOAPZKYY6JZX5WM7PK7PTS6A, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:56.282396896 (432h56m56.153479941s) [noise=true , address=45.91.116.10:4084 , piece_id=HAAU43H7A3UTI5XGHBFF6CEWCKWZ57KW4YTUHZZXUIXTUAN45VZA, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:56.614990237 (432h56m56.486073272s) [noise=true , address=77.90.31.11:4554 , piece_id=MEALTULOBOHZOXE6IA3Q4HXU4UH2DXUOEI2PAROQGTYWT4KGMJVQ, seg_pos={0 0}, upload=380056]: Done uploading piece. err: <nil> Dec 23 16:25:56.615010896 (432h56m56.486093931s) [seg_pos={0 0}, upload=380056]: Segment reached optimal threshold of 13 pieces. Dec 23 16:25:56.615211789 (432h56m56.486294834s) [noise=true , address=145.79.227.40:4050 , piece_id=SI34DQR2C756H4ITMA5Z3MXWS2ZJGCKH3W232BB2OBG3LUWDDGIA, seg_pos={0 0}, upload=380056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Vw9PtVFLQPpm2uGV55tCoK7VfopaBU4EcWNBUyqhNeJjgjG64): protocol: expected piece hash; context canceled; EOF Dec 23 16:25:56.615218632 (432h56m56.486301677s) [noise=true , address=145.79.227.40:4050 , piece_id=SI34DQR2C756H4ITMA5Z3MXWS2ZJGCKH3W232BB2OBG3LUWDDGIA, seg_pos={0 0}, upload=380056]: Long tail cancellation due to optimal threshold Dec 23 16:25:56.615497543 (432h56m56.486580588s) [noise=true , address=45.91.116.14:4191 , piece_id=MJLISQSYJFUGWWK46OTMUZCZ2VQQ4EHS3S7YBFKCD4T3YXMWJVVA, seg_pos={0 0}, upload=380056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: write tcp 188.166.225.170:54656->45.91.116.14:4191: use of closed network connection; write tcp 188.166.225.170:54656->45.91.116.14:4191: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:54656->45.91.116.14:4191: use of closed network connection Dec 23 16:25:56.615517341 (432h56m56.486600376s) [noise=true , address=45.91.116.14:4191 , piece_id=MJLISQSYJFUGWWK46OTMUZCZ2VQQ4EHS3S7YBFKCD4T3YXMWJVVA, seg_pos={0 0}, upload=380056]: Long tail cancellation due to optimal threshold Dec 23 16:25:56.615895681 (432h56m56.486978727s) [noise=true , address=45.91.116.9:4261 , piece_id=EOTTVRXGA3I3CQYZV5IR6N6O4CUEQSQW6WJTCD63DYRFPDPY4BVA, seg_pos={0 0}, upload=380056]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: expected piece hash; context canceled; EOF Dec 23 16:25:56.615903075 (432h56m56.48698611s) [noise=true , address=45.91.116.9:4261 , piece_id=EOTTVRXGA3I3CQYZV5IR6N6O4CUEQSQW6WJTCD63DYRFPDPY4BVA, seg_pos={0 0}, upload=380056]: Long tail cancellation due to optimal threshold Dec 23 16:25:56.615921630 (432h56m56.487004675s) [seg_pos={0 0}, upload=380056]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 23 16:25:56.616054954 (432h56m56.487137989s) [upload=380056]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 23 16:25:56.616061106 (432h56m56.487144151s) [upload=380056]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 23 16:25:56.907724130 (432h56m56.778807165s) [upload=380056]: Done uploading segments... Dec 23 16:25:56.907742444 (432h56m56.778825479s) [upload=380056]: Upload finished. err: <nil> Duration: 5.767999481s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)