Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 5.847140951s trace: https://jaeger.dev.storj.io/trace/3ecb905698746e72 FAIL (2025-12-21 11:02:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: UH4YHD7JFAKXOXJTNKRAUITFZVFWU5AJG5XHSLCJ6JS7JCLWEVRQ) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 3G4VLJ76W4UGBDWZEIQCBDFPRGAM7QL4GMC23LS73BC3JSFGGHPA) context deadline exceeded ======LOGS====== Dec 21 10:27:32.520070050 (378h58m32.391153085s) [upload=332595]: Starting upload Dec 21 10:27:32.520134143 (378h58m32.391217178s) [upload=332595]: Done starting upload Dec 21 10:27:32.520194427 (378h58m32.391277472s) [upload=332595]: Uploading segments... Dec 21 10:27:32.520306451 (378h58m32.391389486s) [upload=332595]: Got next segment. Inline: false Dec 21 10:27:32.520335335 (378h58m32.39141839s) [upload=332595]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 10:27:32.520339694 (378h58m32.391422729s) [upload=332595]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 10:27:32.777396672 (378h58m32.648479718s) [seg_pos={0 0}, upload=332595]: Begin upload segment... Dec 21 10:27:32.777460795 (378h58m32.64854384s) [seg_pos={0 0}, upload=332595]: Done begin upload segment. Dec 21 10:27:32.777515899 (378h58m32.648598934s) [noise=true , address=145.79.227.23:4547 , piece_id=OOD2VMVRTCVGANLHQJME4HCJACBJEH2OUSKZBUY6RVNMSXA3CM7Q, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.777528564 (378h58m32.648611619s) [noise=true , address=145.79.227.27:4691 , piece_id=PEBFRJ6FC4PNFEODQJFGBHHC56RKLUECXWW36YBHNO47HDCDXJWQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.777605250 (378h58m32.648688295s) [noise=true , address=145.79.227.50:4373 , piece_id=GVHEJYGQ6A3HKPZ4V563E37K4L3CZUDEGCVJJHGNC722L3BNY46Q, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.778388482 (378h58m32.649471527s) [noise=true , address=145.79.227.39:4015 , piece_id=QB5S57M5HEYFEZOZYHBWMNFWVLQQEGHS73TBMANOJCCB42LK2AVA, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.778834370 (378h58m32.649917405s) [noise=true , address=45.91.116.12:4225 , piece_id=WH4P44CPP53IJXXMGK5VK2E3PYBGFLJBWSCFSCBKR2QTALIIEGEA, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.778858857 (378h58m32.649941892s) [noise=true , address=145.79.227.42:4121 , piece_id=FA7IZRDCG2XNN6PNDJOTZXGQXZWDOVV2XPR7MUSLK3YVYJVZE4UQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779235654 (378h58m32.650318689s) [noise=true , address=45.91.116.7:4011 , piece_id=UH4YHD7JFAKXOXJTNKRAUITFZVFWU5AJG5XHSLCJ6JS7JCLWEVRQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779269559 (378h58m32.650352594s) [noise=true , address=45.91.116.4:4940 , piece_id=L6QW3XENG6UBHKUUXTMOQQRIXBWBN6XXL7SHLY7W5KC5ES3SFS3A, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779536989 (378h58m32.650620024s) [noise=true , address=145.79.227.52:4445 , piece_id=YWZ6UHBVJWXLIGSE5HMV4WWZLLQLMBCB35WDSUFXK4XMBU5ZAT7Q, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779807954 (378h58m32.650890999s) [noise=true , address=77.90.31.15:4700 , piece_id=WQP7T5QHGW23C7I5TBY7PESM5YTTXS3BAEH3RDRQTJZYHI37BUJQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.780102345 (378h58m32.65118538s) [noise=true , address=145.79.227.17:4972 , piece_id=B3KYU5P3RL2BFR7QTS5ZK7B6UBDB3DTNTQKAGPM7B75S2SYEUFBQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.780482589 (378h58m32.651565634s) [noise=true , address=45.91.116.8:4048 , piece_id=3G4VLJ76W4UGBDWZEIQCBDFPRGAM7QL4GMC23LS73BC3JSFGGHPA, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.780863915 (378h58m32.65194696s) [noise=true , address=145.79.227.48:4303 , piece_id=44PNGUZODJGAD7AFPZGJE6XBDYCSN2CLPIARGZI62PQ7KKU7ANWQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.781783866 (378h58m32.652866911s) [noise=true , address=145.79.227.35:4482 , piece_id=3FUQ4OSF4B5DNWTAPY5W3LS2ZFBLCDWGXXLGASBKACSX3ONKLWWQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.782593317 (378h58m32.653676393s) [noise=true , address=45.91.116.14:4189 , piece_id=NF2SZ7V2LDV76ADGQ6NXVNBHTHNLMRYS4SXW6BT4OM7BQ5TQ6BPQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.783214190 (378h58m32.654297225s) [noise=true , address=145.79.227.51:4411 , piece_id=LC6PCUVNJN77OICI5JGXWN7LJD6JGMKKHHIQG6T4MKX5SU7FGRIQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:34.478418529 (378h58m34.349501564s) [upload=332595]: Next returned nil segment Dec 21 10:27:34.478430381 (378h58m34.349513416s) [upload=332595]: Waiting for error group managing segments... Dec 21 10:27:34.763620982 (378h58m34.634704027s) [noise=true , address=145.79.227.51:4411 , piece_id=LC6PCUVNJN77OICI5JGXWN7LJD6JGMKKHHIQG6T4MKX5SU7FGRIQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.770992478 (378h58m34.642075514s) [noise=true , address=145.79.227.50:4373 , piece_id=GVHEJYGQ6A3HKPZ4V563E37K4L3CZUDEGCVJJHGNC722L3BNY46Q, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.776530073 (378h58m34.647613108s) [noise=true , address=145.79.227.42:4121 , piece_id=FA7IZRDCG2XNN6PNDJOTZXGQXZWDOVV2XPR7MUSLK3YVYJVZE4UQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.786521177 (378h58m34.657604222s) [noise=true , address=145.79.227.17:4972 , piece_id=B3KYU5P3RL2BFR7QTS5ZK7B6UBDB3DTNTQKAGPM7B75S2SYEUFBQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.786591240 (378h58m34.657674275s) [noise=true , address=145.79.227.23:4547 , piece_id=OOD2VMVRTCVGANLHQJME4HCJACBJEH2OUSKZBUY6RVNMSXA3CM7Q, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.787618627 (378h58m34.658701672s) [noise=true , address=145.79.227.52:4445 , piece_id=YWZ6UHBVJWXLIGSE5HMV4WWZLLQLMBCB35WDSUFXK4XMBU5ZAT7Q, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.794650276 (378h58m34.665733311s) [noise=true , address=145.79.227.35:4482 , piece_id=3FUQ4OSF4B5DNWTAPY5W3LS2ZFBLCDWGXXLGASBKACSX3ONKLWWQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.794701083 (378h58m34.665784128s) [noise=true , address=145.79.227.39:4015 , piece_id=QB5S57M5HEYFEZOZYHBWMNFWVLQQEGHS73TBMANOJCCB42LK2AVA, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.833604161 (378h58m34.704687196s) [noise=true , address=145.79.227.48:4303 , piece_id=44PNGUZODJGAD7AFPZGJE6XBDYCSN2CLPIARGZI62PQ7KKU7ANWQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.857890877 (378h58m34.728973912s) [noise=true , address=145.79.227.27:4691 , piece_id=PEBFRJ6FC4PNFEODQJFGBHHC56RKLUECXWW36YBHNO47HDCDXJWQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.991414468 (378h58m34.862497503s) [noise=true , address=45.91.116.7:4011 , piece_id=UH4YHD7JFAKXOXJTNKRAUITFZVFWU5AJG5XHSLCJ6JS7JCLWEVRQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:35.048985192 (378h58m34.920068237s) [noise=true , address=45.91.116.14:4189 , piece_id=NF2SZ7V2LDV76ADGQ6NXVNBHTHNLMRYS4SXW6BT4OM7BQ5TQ6BPQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:35.055578377 (378h58m34.926661412s) [noise=true , address=45.91.116.8:4048 , piece_id=3G4VLJ76W4UGBDWZEIQCBDFPRGAM7QL4GMC23LS73BC3JSFGGHPA, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:35.055594557 (378h58m34.926677602s) [seg_pos={0 0}, upload=332595]: Segment reached optimal threshold of 13 pieces. Dec 21 10:27:35.055773338 (378h58m34.926856373s) [noise=true , address=45.91.116.4:4940 , piece_id=L6QW3XENG6UBHKUUXTMOQQRIXBWBN6XXL7SHLY7W5KC5ES3SFS3A, seg_pos={0 0}, upload=332595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 21 10:27:35.055779740 (378h58m34.926862775s) [noise=true , address=45.91.116.4:4940 , piece_id=L6QW3XENG6UBHKUUXTMOQQRIXBWBN6XXL7SHLY7W5KC5ES3SFS3A, seg_pos={0 0}, upload=332595]: Long tail cancellation due to optimal threshold Dec 21 10:27:35.055834805 (378h58m34.92691785s) [noise=true , address=77.90.31.15:4700 , piece_id=WQP7T5QHGW23C7I5TBY7PESM5YTTXS3BAEH3RDRQTJZYHI37BUJQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 21 10:27:35.055845194 (378h58m34.926928239s) [noise=true , address=77.90.31.15:4700 , piece_id=WQP7T5QHGW23C7I5TBY7PESM5YTTXS3BAEH3RDRQTJZYHI37BUJQ, seg_pos={0 0}, upload=332595]: Long tail cancellation due to optimal threshold Dec 21 10:27:35.055982826 (378h58m34.927065862s) [noise=true , address=45.91.116.12:4225 , piece_id=WH4P44CPP53IJXXMGK5VK2E3PYBGFLJBWSCFSCBKR2QTALIIEGEA, seg_pos={0 0}, upload=332595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 21 10:27:35.056041539 (378h58m34.927124594s) [noise=true , address=45.91.116.12:4225 , piece_id=WH4P44CPP53IJXXMGK5VK2E3PYBGFLJBWSCFSCBKR2QTALIIEGEA, seg_pos={0 0}, upload=332595]: Long tail cancellation due to optimal threshold Dec 21 10:27:35.056064683 (378h58m34.927147728s) [seg_pos={0 0}, upload=332595]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 10:27:35.056146969 (378h58m34.927230004s) [upload=332595]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 10:27:35.056153121 (378h58m34.927236176s) [upload=332595]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 10:27:35.353226207 (378h58m35.224309242s) [upload=332595]: Done uploading segments... Dec 21 10:27:35.353243942 (378h58m35.224326977s) [upload=332595]: Upload finished. err: <nil> Duration: 5.847169856s core uplink us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 766.363065ms trace: https://jaeger.dev.storj.io/trace/37011901b11802c9 FAIL (2025-12-21 11:01:52 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:37524->45.38.24.2:443: read: connection reset by peer Duration: 766.391701ms edge authservice http us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 5.697821001s trace: https://jaeger.dev.storj.io/trace/3ecb905698746e72 FAIL (2025-12-21 11:01:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: UH4YHD7JFAKXOXJTNKRAUITFZVFWU5AJG5XHSLCJ6JS7JCLWEVRQ) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 3G4VLJ76W4UGBDWZEIQCBDFPRGAM7QL4GMC23LS73BC3JSFGGHPA) context deadline exceeded ======LOGS====== Dec 21 10:27:32.520070050 (378h58m32.391153085s) [upload=332595]: Starting upload Dec 21 10:27:32.520134143 (378h58m32.391217178s) [upload=332595]: Done starting upload Dec 21 10:27:32.520194427 (378h58m32.391277472s) [upload=332595]: Uploading segments... Dec 21 10:27:32.520306451 (378h58m32.391389486s) [upload=332595]: Got next segment. Inline: false Dec 21 10:27:32.520335335 (378h58m32.39141839s) [upload=332595]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 10:27:32.520339694 (378h58m32.391422729s) [upload=332595]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 10:27:32.777396672 (378h58m32.648479718s) [seg_pos={0 0}, upload=332595]: Begin upload segment... Dec 21 10:27:32.777460795 (378h58m32.64854384s) [seg_pos={0 0}, upload=332595]: Done begin upload segment. Dec 21 10:27:32.777515899 (378h58m32.648598934s) [noise=true , address=145.79.227.23:4547 , piece_id=OOD2VMVRTCVGANLHQJME4HCJACBJEH2OUSKZBUY6RVNMSXA3CM7Q, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.777528564 (378h58m32.648611619s) [noise=true , address=145.79.227.27:4691 , piece_id=PEBFRJ6FC4PNFEODQJFGBHHC56RKLUECXWW36YBHNO47HDCDXJWQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.777605250 (378h58m32.648688295s) [noise=true , address=145.79.227.50:4373 , piece_id=GVHEJYGQ6A3HKPZ4V563E37K4L3CZUDEGCVJJHGNC722L3BNY46Q, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.778388482 (378h58m32.649471527s) [noise=true , address=145.79.227.39:4015 , piece_id=QB5S57M5HEYFEZOZYHBWMNFWVLQQEGHS73TBMANOJCCB42LK2AVA, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.778834370 (378h58m32.649917405s) [noise=true , address=45.91.116.12:4225 , piece_id=WH4P44CPP53IJXXMGK5VK2E3PYBGFLJBWSCFSCBKR2QTALIIEGEA, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.778858857 (378h58m32.649941892s) [noise=true , address=145.79.227.42:4121 , piece_id=FA7IZRDCG2XNN6PNDJOTZXGQXZWDOVV2XPR7MUSLK3YVYJVZE4UQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779235654 (378h58m32.650318689s) [noise=true , address=45.91.116.7:4011 , piece_id=UH4YHD7JFAKXOXJTNKRAUITFZVFWU5AJG5XHSLCJ6JS7JCLWEVRQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779269559 (378h58m32.650352594s) [noise=true , address=45.91.116.4:4940 , piece_id=L6QW3XENG6UBHKUUXTMOQQRIXBWBN6XXL7SHLY7W5KC5ES3SFS3A, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779536989 (378h58m32.650620024s) [noise=true , address=145.79.227.52:4445 , piece_id=YWZ6UHBVJWXLIGSE5HMV4WWZLLQLMBCB35WDSUFXK4XMBU5ZAT7Q, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779807954 (378h58m32.650890999s) [noise=true , address=77.90.31.15:4700 , piece_id=WQP7T5QHGW23C7I5TBY7PESM5YTTXS3BAEH3RDRQTJZYHI37BUJQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.780102345 (378h58m32.65118538s) [noise=true , address=145.79.227.17:4972 , piece_id=B3KYU5P3RL2BFR7QTS5ZK7B6UBDB3DTNTQKAGPM7B75S2SYEUFBQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.780482589 (378h58m32.651565634s) [noise=true , address=45.91.116.8:4048 , piece_id=3G4VLJ76W4UGBDWZEIQCBDFPRGAM7QL4GMC23LS73BC3JSFGGHPA, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.780863915 (378h58m32.65194696s) [noise=true , address=145.79.227.48:4303 , piece_id=44PNGUZODJGAD7AFPZGJE6XBDYCSN2CLPIARGZI62PQ7KKU7ANWQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.781783866 (378h58m32.652866911s) [noise=true , address=145.79.227.35:4482 , piece_id=3FUQ4OSF4B5DNWTAPY5W3LS2ZFBLCDWGXXLGASBKACSX3ONKLWWQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.782593317 (378h58m32.653676393s) [noise=true , address=45.91.116.14:4189 , piece_id=NF2SZ7V2LDV76ADGQ6NXVNBHTHNLMRYS4SXW6BT4OM7BQ5TQ6BPQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.783214190 (378h58m32.654297225s) [noise=true , address=145.79.227.51:4411 , piece_id=LC6PCUVNJN77OICI5JGXWN7LJD6JGMKKHHIQG6T4MKX5SU7FGRIQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:34.478418529 (378h58m34.349501564s) [upload=332595]: Next returned nil segment Dec 21 10:27:34.478430381 (378h58m34.349513416s) [upload=332595]: Waiting for error group managing segments... Dec 21 10:27:34.763620982 (378h58m34.634704027s) [noise=true , address=145.79.227.51:4411 , piece_id=LC6PCUVNJN77OICI5JGXWN7LJD6JGMKKHHIQG6T4MKX5SU7FGRIQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.770992478 (378h58m34.642075514s) [noise=true , address=145.79.227.50:4373 , piece_id=GVHEJYGQ6A3HKPZ4V563E37K4L3CZUDEGCVJJHGNC722L3BNY46Q, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.776530073 (378h58m34.647613108s) [noise=true , address=145.79.227.42:4121 , piece_id=FA7IZRDCG2XNN6PNDJOTZXGQXZWDOVV2XPR7MUSLK3YVYJVZE4UQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.786521177 (378h58m34.657604222s) [noise=true , address=145.79.227.17:4972 , piece_id=B3KYU5P3RL2BFR7QTS5ZK7B6UBDB3DTNTQKAGPM7B75S2SYEUFBQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.786591240 (378h58m34.657674275s) [noise=true , address=145.79.227.23:4547 , piece_id=OOD2VMVRTCVGANLHQJME4HCJACBJEH2OUSKZBUY6RVNMSXA3CM7Q, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.787618627 (378h58m34.658701672s) [noise=true , address=145.79.227.52:4445 , piece_id=YWZ6UHBVJWXLIGSE5HMV4WWZLLQLMBCB35WDSUFXK4XMBU5ZAT7Q, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.794650276 (378h58m34.665733311s) [noise=true , address=145.79.227.35:4482 , piece_id=3FUQ4OSF4B5DNWTAPY5W3LS2ZFBLCDWGXXLGASBKACSX3ONKLWWQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.794701083 (378h58m34.665784128s) [noise=true , address=145.79.227.39:4015 , piece_id=QB5S57M5HEYFEZOZYHBWMNFWVLQQEGHS73TBMANOJCCB42LK2AVA, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.833604161 (378h58m34.704687196s) [noise=true , address=145.79.227.48:4303 , piece_id=44PNGUZODJGAD7AFPZGJE6XBDYCSN2CLPIARGZI62PQ7KKU7ANWQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.857890877 (378h58m34.728973912s) [noise=true , address=145.79.227.27:4691 , piece_id=PEBFRJ6FC4PNFEODQJFGBHHC56RKLUECXWW36YBHNO47HDCDXJWQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.991414468 (378h58m34.862497503s) [noise=true , address=45.91.116.7:4011 , piece_id=UH4YHD7JFAKXOXJTNKRAUITFZVFWU5AJG5XHSLCJ6JS7JCLWEVRQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:35.048985192 (378h58m34.920068237s) [noise=true , address=45.91.116.14:4189 , piece_id=NF2SZ7V2LDV76ADGQ6NXVNBHTHNLMRYS4SXW6BT4OM7BQ5TQ6BPQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:35.055578377 (378h58m34.926661412s) [noise=true , address=45.91.116.8:4048 , piece_id=3G4VLJ76W4UGBDWZEIQCBDFPRGAM7QL4GMC23LS73BC3JSFGGHPA, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:35.055594557 (378h58m34.926677602s) [seg_pos={0 0}, upload=332595]: Segment reached optimal threshold of 13 pieces. Dec 21 10:27:35.055773338 (378h58m34.926856373s) [noise=true , address=45.91.116.4:4940 , piece_id=L6QW3XENG6UBHKUUXTMOQQRIXBWBN6XXL7SHLY7W5KC5ES3SFS3A, seg_pos={0 0}, upload=332595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 21 10:27:35.055779740 (378h58m34.926862775s) [noise=true , address=45.91.116.4:4940 , piece_id=L6QW3XENG6UBHKUUXTMOQQRIXBWBN6XXL7SHLY7W5KC5ES3SFS3A, seg_pos={0 0}, upload=332595]: Long tail cancellation due to optimal threshold Dec 21 10:27:35.055834805 (378h58m34.92691785s) [noise=true , address=77.90.31.15:4700 , piece_id=WQP7T5QHGW23C7I5TBY7PESM5YTTXS3BAEH3RDRQTJZYHI37BUJQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 21 10:27:35.055845194 (378h58m34.926928239s) [noise=true , address=77.90.31.15:4700 , piece_id=WQP7T5QHGW23C7I5TBY7PESM5YTTXS3BAEH3RDRQTJZYHI37BUJQ, seg_pos={0 0}, upload=332595]: Long tail cancellation due to optimal threshold Dec 21 10:27:35.055982826 (378h58m34.927065862s) [noise=true , address=45.91.116.12:4225 , piece_id=WH4P44CPP53IJXXMGK5VK2E3PYBGFLJBWSCFSCBKR2QTALIIEGEA, seg_pos={0 0}, upload=332595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 21 10:27:35.056041539 (378h58m34.927124594s) [noise=true , address=45.91.116.12:4225 , piece_id=WH4P44CPP53IJXXMGK5VK2E3PYBGFLJBWSCFSCBKR2QTALIIEGEA, seg_pos={0 0}, upload=332595]: Long tail cancellation due to optimal threshold Dec 21 10:27:35.056064683 (378h58m34.927147728s) [seg_pos={0 0}, upload=332595]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 10:27:35.056146969 (378h58m34.927230004s) [upload=332595]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 10:27:35.056153121 (378h58m34.927236176s) [upload=332595]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 10:27:35.353226207 (378h58m35.224309242s) [upload=332595]: Done uploading segments... Dec 21 10:27:35.353243942 (378h58m35.224326977s) [upload=332595]: Upload finished. err: <nil> Duration: 5.697847702s core uplink us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 768.172641ms trace: https://jaeger.dev.storj.io/trace/37011901b11802c9 FAIL (2025-12-21 11:01:03 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:37524->45.38.24.2:443: read: connection reset by peer Duration: 768.202249ms edge authservice http us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 5.848146227s trace: https://jaeger.dev.storj.io/trace/3ecb905698746e72 FAIL (2025-12-21 11:00:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: UH4YHD7JFAKXOXJTNKRAUITFZVFWU5AJG5XHSLCJ6JS7JCLWEVRQ) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 3G4VLJ76W4UGBDWZEIQCBDFPRGAM7QL4GMC23LS73BC3JSFGGHPA) context deadline exceeded ======LOGS====== Dec 21 10:27:32.520070050 (378h58m32.391153085s) [upload=332595]: Starting upload Dec 21 10:27:32.520134143 (378h58m32.391217178s) [upload=332595]: Done starting upload Dec 21 10:27:32.520194427 (378h58m32.391277472s) [upload=332595]: Uploading segments... Dec 21 10:27:32.520306451 (378h58m32.391389486s) [upload=332595]: Got next segment. Inline: false Dec 21 10:27:32.520335335 (378h58m32.39141839s) [upload=332595]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 10:27:32.520339694 (378h58m32.391422729s) [upload=332595]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 10:27:32.777396672 (378h58m32.648479718s) [seg_pos={0 0}, upload=332595]: Begin upload segment... Dec 21 10:27:32.777460795 (378h58m32.64854384s) [seg_pos={0 0}, upload=332595]: Done begin upload segment. Dec 21 10:27:32.777515899 (378h58m32.648598934s) [noise=true , address=145.79.227.23:4547 , piece_id=OOD2VMVRTCVGANLHQJME4HCJACBJEH2OUSKZBUY6RVNMSXA3CM7Q, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.777528564 (378h58m32.648611619s) [noise=true , address=145.79.227.27:4691 , piece_id=PEBFRJ6FC4PNFEODQJFGBHHC56RKLUECXWW36YBHNO47HDCDXJWQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.777605250 (378h58m32.648688295s) [noise=true , address=145.79.227.50:4373 , piece_id=GVHEJYGQ6A3HKPZ4V563E37K4L3CZUDEGCVJJHGNC722L3BNY46Q, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.778388482 (378h58m32.649471527s) [noise=true , address=145.79.227.39:4015 , piece_id=QB5S57M5HEYFEZOZYHBWMNFWVLQQEGHS73TBMANOJCCB42LK2AVA, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.778834370 (378h58m32.649917405s) [noise=true , address=45.91.116.12:4225 , piece_id=WH4P44CPP53IJXXMGK5VK2E3PYBGFLJBWSCFSCBKR2QTALIIEGEA, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.778858857 (378h58m32.649941892s) [noise=true , address=145.79.227.42:4121 , piece_id=FA7IZRDCG2XNN6PNDJOTZXGQXZWDOVV2XPR7MUSLK3YVYJVZE4UQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779235654 (378h58m32.650318689s) [noise=true , address=45.91.116.7:4011 , piece_id=UH4YHD7JFAKXOXJTNKRAUITFZVFWU5AJG5XHSLCJ6JS7JCLWEVRQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779269559 (378h58m32.650352594s) [noise=true , address=45.91.116.4:4940 , piece_id=L6QW3XENG6UBHKUUXTMOQQRIXBWBN6XXL7SHLY7W5KC5ES3SFS3A, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779536989 (378h58m32.650620024s) [noise=true , address=145.79.227.52:4445 , piece_id=YWZ6UHBVJWXLIGSE5HMV4WWZLLQLMBCB35WDSUFXK4XMBU5ZAT7Q, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779807954 (378h58m32.650890999s) [noise=true , address=77.90.31.15:4700 , piece_id=WQP7T5QHGW23C7I5TBY7PESM5YTTXS3BAEH3RDRQTJZYHI37BUJQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.780102345 (378h58m32.65118538s) [noise=true , address=145.79.227.17:4972 , piece_id=B3KYU5P3RL2BFR7QTS5ZK7B6UBDB3DTNTQKAGPM7B75S2SYEUFBQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.780482589 (378h58m32.651565634s) [noise=true , address=45.91.116.8:4048 , piece_id=3G4VLJ76W4UGBDWZEIQCBDFPRGAM7QL4GMC23LS73BC3JSFGGHPA, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.780863915 (378h58m32.65194696s) [noise=true , address=145.79.227.48:4303 , piece_id=44PNGUZODJGAD7AFPZGJE6XBDYCSN2CLPIARGZI62PQ7KKU7ANWQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.781783866 (378h58m32.652866911s) [noise=true , address=145.79.227.35:4482 , piece_id=3FUQ4OSF4B5DNWTAPY5W3LS2ZFBLCDWGXXLGASBKACSX3ONKLWWQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.782593317 (378h58m32.653676393s) [noise=true , address=45.91.116.14:4189 , piece_id=NF2SZ7V2LDV76ADGQ6NXVNBHTHNLMRYS4SXW6BT4OM7BQ5TQ6BPQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.783214190 (378h58m32.654297225s) [noise=true , address=145.79.227.51:4411 , piece_id=LC6PCUVNJN77OICI5JGXWN7LJD6JGMKKHHIQG6T4MKX5SU7FGRIQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:34.478418529 (378h58m34.349501564s) [upload=332595]: Next returned nil segment Dec 21 10:27:34.478430381 (378h58m34.349513416s) [upload=332595]: Waiting for error group managing segments... Dec 21 10:27:34.763620982 (378h58m34.634704027s) [noise=true , address=145.79.227.51:4411 , piece_id=LC6PCUVNJN77OICI5JGXWN7LJD6JGMKKHHIQG6T4MKX5SU7FGRIQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.770992478 (378h58m34.642075514s) [noise=true , address=145.79.227.50:4373 , piece_id=GVHEJYGQ6A3HKPZ4V563E37K4L3CZUDEGCVJJHGNC722L3BNY46Q, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.776530073 (378h58m34.647613108s) [noise=true , address=145.79.227.42:4121 , piece_id=FA7IZRDCG2XNN6PNDJOTZXGQXZWDOVV2XPR7MUSLK3YVYJVZE4UQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.786521177 (378h58m34.657604222s) [noise=true , address=145.79.227.17:4972 , piece_id=B3KYU5P3RL2BFR7QTS5ZK7B6UBDB3DTNTQKAGPM7B75S2SYEUFBQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.786591240 (378h58m34.657674275s) [noise=true , address=145.79.227.23:4547 , piece_id=OOD2VMVRTCVGANLHQJME4HCJACBJEH2OUSKZBUY6RVNMSXA3CM7Q, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.787618627 (378h58m34.658701672s) [noise=true , address=145.79.227.52:4445 , piece_id=YWZ6UHBVJWXLIGSE5HMV4WWZLLQLMBCB35WDSUFXK4XMBU5ZAT7Q, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.794650276 (378h58m34.665733311s) [noise=true , address=145.79.227.35:4482 , piece_id=3FUQ4OSF4B5DNWTAPY5W3LS2ZFBLCDWGXXLGASBKACSX3ONKLWWQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.794701083 (378h58m34.665784128s) [noise=true , address=145.79.227.39:4015 , piece_id=QB5S57M5HEYFEZOZYHBWMNFWVLQQEGHS73TBMANOJCCB42LK2AVA, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.833604161 (378h58m34.704687196s) [noise=true , address=145.79.227.48:4303 , piece_id=44PNGUZODJGAD7AFPZGJE6XBDYCSN2CLPIARGZI62PQ7KKU7ANWQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.857890877 (378h58m34.728973912s) [noise=true , address=145.79.227.27:4691 , piece_id=PEBFRJ6FC4PNFEODQJFGBHHC56RKLUECXWW36YBHNO47HDCDXJWQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.991414468 (378h58m34.862497503s) [noise=true , address=45.91.116.7:4011 , piece_id=UH4YHD7JFAKXOXJTNKRAUITFZVFWU5AJG5XHSLCJ6JS7JCLWEVRQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:35.048985192 (378h58m34.920068237s) [noise=true , address=45.91.116.14:4189 , piece_id=NF2SZ7V2LDV76ADGQ6NXVNBHTHNLMRYS4SXW6BT4OM7BQ5TQ6BPQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:35.055578377 (378h58m34.926661412s) [noise=true , address=45.91.116.8:4048 , piece_id=3G4VLJ76W4UGBDWZEIQCBDFPRGAM7QL4GMC23LS73BC3JSFGGHPA, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:35.055594557 (378h58m34.926677602s) [seg_pos={0 0}, upload=332595]: Segment reached optimal threshold of 13 pieces. Dec 21 10:27:35.055773338 (378h58m34.926856373s) [noise=true , address=45.91.116.4:4940 , piece_id=L6QW3XENG6UBHKUUXTMOQQRIXBWBN6XXL7SHLY7W5KC5ES3SFS3A, seg_pos={0 0}, upload=332595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 21 10:27:35.055779740 (378h58m34.926862775s) [noise=true , address=45.91.116.4:4940 , piece_id=L6QW3XENG6UBHKUUXTMOQQRIXBWBN6XXL7SHLY7W5KC5ES3SFS3A, seg_pos={0 0}, upload=332595]: Long tail cancellation due to optimal threshold Dec 21 10:27:35.055834805 (378h58m34.92691785s) [noise=true , address=77.90.31.15:4700 , piece_id=WQP7T5QHGW23C7I5TBY7PESM5YTTXS3BAEH3RDRQTJZYHI37BUJQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 21 10:27:35.055845194 (378h58m34.926928239s) [noise=true , address=77.90.31.15:4700 , piece_id=WQP7T5QHGW23C7I5TBY7PESM5YTTXS3BAEH3RDRQTJZYHI37BUJQ, seg_pos={0 0}, upload=332595]: Long tail cancellation due to optimal threshold Dec 21 10:27:35.055982826 (378h58m34.927065862s) [noise=true , address=45.91.116.12:4225 , piece_id=WH4P44CPP53IJXXMGK5VK2E3PYBGFLJBWSCFSCBKR2QTALIIEGEA, seg_pos={0 0}, upload=332595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 21 10:27:35.056041539 (378h58m34.927124594s) [noise=true , address=45.91.116.12:4225 , piece_id=WH4P44CPP53IJXXMGK5VK2E3PYBGFLJBWSCFSCBKR2QTALIIEGEA, seg_pos={0 0}, upload=332595]: Long tail cancellation due to optimal threshold Dec 21 10:27:35.056064683 (378h58m34.927147728s) [seg_pos={0 0}, upload=332595]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 10:27:35.056146969 (378h58m34.927230004s) [upload=332595]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 10:27:35.056153121 (378h58m34.927236176s) [upload=332595]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 10:27:35.353226207 (378h58m35.224309242s) [upload=332595]: Done uploading segments... Dec 21 10:27:35.353243942 (378h58m35.224326977s) [upload=332595]: Upload finished. err: <nil> Duration: 5.84817374s core uplink us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 808.580613ms trace: https://jaeger.dev.storj.io/trace/37011901b11802c9 FAIL (2025-12-21 11:00:25 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:37524->45.38.24.2:443: read: connection reset by peer Duration: 808.61072ms edge authservice http us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.861882545s trace: https://jaeger.dev.storj.io/trace/3ecb905698746e72 FAIL (2025-12-21 11:00:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: UH4YHD7JFAKXOXJTNKRAUITFZVFWU5AJG5XHSLCJ6JS7JCLWEVRQ) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 3G4VLJ76W4UGBDWZEIQCBDFPRGAM7QL4GMC23LS73BC3JSFGGHPA) context deadline exceeded ======LOGS====== Dec 21 10:27:32.520070050 (378h58m32.391153085s) [upload=332595]: Starting upload Dec 21 10:27:32.520134143 (378h58m32.391217178s) [upload=332595]: Done starting upload Dec 21 10:27:32.520194427 (378h58m32.391277472s) [upload=332595]: Uploading segments... Dec 21 10:27:32.520306451 (378h58m32.391389486s) [upload=332595]: Got next segment. Inline: false Dec 21 10:27:32.520335335 (378h58m32.39141839s) [upload=332595]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 10:27:32.520339694 (378h58m32.391422729s) [upload=332595]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 10:27:32.777396672 (378h58m32.648479718s) [seg_pos={0 0}, upload=332595]: Begin upload segment... Dec 21 10:27:32.777460795 (378h58m32.64854384s) [seg_pos={0 0}, upload=332595]: Done begin upload segment. Dec 21 10:27:32.777515899 (378h58m32.648598934s) [noise=true , address=145.79.227.23:4547 , piece_id=OOD2VMVRTCVGANLHQJME4HCJACBJEH2OUSKZBUY6RVNMSXA3CM7Q, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.777528564 (378h58m32.648611619s) [noise=true , address=145.79.227.27:4691 , piece_id=PEBFRJ6FC4PNFEODQJFGBHHC56RKLUECXWW36YBHNO47HDCDXJWQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.777605250 (378h58m32.648688295s) [noise=true , address=145.79.227.50:4373 , piece_id=GVHEJYGQ6A3HKPZ4V563E37K4L3CZUDEGCVJJHGNC722L3BNY46Q, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.778388482 (378h58m32.649471527s) [noise=true , address=145.79.227.39:4015 , piece_id=QB5S57M5HEYFEZOZYHBWMNFWVLQQEGHS73TBMANOJCCB42LK2AVA, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.778834370 (378h58m32.649917405s) [noise=true , address=45.91.116.12:4225 , piece_id=WH4P44CPP53IJXXMGK5VK2E3PYBGFLJBWSCFSCBKR2QTALIIEGEA, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.778858857 (378h58m32.649941892s) [noise=true , address=145.79.227.42:4121 , piece_id=FA7IZRDCG2XNN6PNDJOTZXGQXZWDOVV2XPR7MUSLK3YVYJVZE4UQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779235654 (378h58m32.650318689s) [noise=true , address=45.91.116.7:4011 , piece_id=UH4YHD7JFAKXOXJTNKRAUITFZVFWU5AJG5XHSLCJ6JS7JCLWEVRQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779269559 (378h58m32.650352594s) [noise=true , address=45.91.116.4:4940 , piece_id=L6QW3XENG6UBHKUUXTMOQQRIXBWBN6XXL7SHLY7W5KC5ES3SFS3A, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779536989 (378h58m32.650620024s) [noise=true , address=145.79.227.52:4445 , piece_id=YWZ6UHBVJWXLIGSE5HMV4WWZLLQLMBCB35WDSUFXK4XMBU5ZAT7Q, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779807954 (378h58m32.650890999s) [noise=true , address=77.90.31.15:4700 , piece_id=WQP7T5QHGW23C7I5TBY7PESM5YTTXS3BAEH3RDRQTJZYHI37BUJQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.780102345 (378h58m32.65118538s) [noise=true , address=145.79.227.17:4972 , piece_id=B3KYU5P3RL2BFR7QTS5ZK7B6UBDB3DTNTQKAGPM7B75S2SYEUFBQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.780482589 (378h58m32.651565634s) [noise=true , address=45.91.116.8:4048 , piece_id=3G4VLJ76W4UGBDWZEIQCBDFPRGAM7QL4GMC23LS73BC3JSFGGHPA, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.780863915 (378h58m32.65194696s) [noise=true , address=145.79.227.48:4303 , piece_id=44PNGUZODJGAD7AFPZGJE6XBDYCSN2CLPIARGZI62PQ7KKU7ANWQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.781783866 (378h58m32.652866911s) [noise=true , address=145.79.227.35:4482 , piece_id=3FUQ4OSF4B5DNWTAPY5W3LS2ZFBLCDWGXXLGASBKACSX3ONKLWWQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.782593317 (378h58m32.653676393s) [noise=true , address=45.91.116.14:4189 , piece_id=NF2SZ7V2LDV76ADGQ6NXVNBHTHNLMRYS4SXW6BT4OM7BQ5TQ6BPQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.783214190 (378h58m32.654297225s) [noise=true , address=145.79.227.51:4411 , piece_id=LC6PCUVNJN77OICI5JGXWN7LJD6JGMKKHHIQG6T4MKX5SU7FGRIQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:34.478418529 (378h58m34.349501564s) [upload=332595]: Next returned nil segment Dec 21 10:27:34.478430381 (378h58m34.349513416s) [upload=332595]: Waiting for error group managing segments... Dec 21 10:27:34.763620982 (378h58m34.634704027s) [noise=true , address=145.79.227.51:4411 , piece_id=LC6PCUVNJN77OICI5JGXWN7LJD6JGMKKHHIQG6T4MKX5SU7FGRIQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.770992478 (378h58m34.642075514s) [noise=true , address=145.79.227.50:4373 , piece_id=GVHEJYGQ6A3HKPZ4V563E37K4L3CZUDEGCVJJHGNC722L3BNY46Q, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.776530073 (378h58m34.647613108s) [noise=true , address=145.79.227.42:4121 , piece_id=FA7IZRDCG2XNN6PNDJOTZXGQXZWDOVV2XPR7MUSLK3YVYJVZE4UQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.786521177 (378h58m34.657604222s) [noise=true , address=145.79.227.17:4972 , piece_id=B3KYU5P3RL2BFR7QTS5ZK7B6UBDB3DTNTQKAGPM7B75S2SYEUFBQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.786591240 (378h58m34.657674275s) [noise=true , address=145.79.227.23:4547 , piece_id=OOD2VMVRTCVGANLHQJME4HCJACBJEH2OUSKZBUY6RVNMSXA3CM7Q, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.787618627 (378h58m34.658701672s) [noise=true , address=145.79.227.52:4445 , piece_id=YWZ6UHBVJWXLIGSE5HMV4WWZLLQLMBCB35WDSUFXK4XMBU5ZAT7Q, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.794650276 (378h58m34.665733311s) [noise=true , address=145.79.227.35:4482 , piece_id=3FUQ4OSF4B5DNWTAPY5W3LS2ZFBLCDWGXXLGASBKACSX3ONKLWWQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.794701083 (378h58m34.665784128s) [noise=true , address=145.79.227.39:4015 , piece_id=QB5S57M5HEYFEZOZYHBWMNFWVLQQEGHS73TBMANOJCCB42LK2AVA, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.833604161 (378h58m34.704687196s) [noise=true , address=145.79.227.48:4303 , piece_id=44PNGUZODJGAD7AFPZGJE6XBDYCSN2CLPIARGZI62PQ7KKU7ANWQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.857890877 (378h58m34.728973912s) [noise=true , address=145.79.227.27:4691 , piece_id=PEBFRJ6FC4PNFEODQJFGBHHC56RKLUECXWW36YBHNO47HDCDXJWQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.991414468 (378h58m34.862497503s) [noise=true , address=45.91.116.7:4011 , piece_id=UH4YHD7JFAKXOXJTNKRAUITFZVFWU5AJG5XHSLCJ6JS7JCLWEVRQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:35.048985192 (378h58m34.920068237s) [noise=true , address=45.91.116.14:4189 , piece_id=NF2SZ7V2LDV76ADGQ6NXVNBHTHNLMRYS4SXW6BT4OM7BQ5TQ6BPQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:35.055578377 (378h58m34.926661412s) [noise=true , address=45.91.116.8:4048 , piece_id=3G4VLJ76W4UGBDWZEIQCBDFPRGAM7QL4GMC23LS73BC3JSFGGHPA, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:35.055594557 (378h58m34.926677602s) [seg_pos={0 0}, upload=332595]: Segment reached optimal threshold of 13 pieces. Dec 21 10:27:35.055773338 (378h58m34.926856373s) [noise=true , address=45.91.116.4:4940 , piece_id=L6QW3XENG6UBHKUUXTMOQQRIXBWBN6XXL7SHLY7W5KC5ES3SFS3A, seg_pos={0 0}, upload=332595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 21 10:27:35.055779740 (378h58m34.926862775s) [noise=true , address=45.91.116.4:4940 , piece_id=L6QW3XENG6UBHKUUXTMOQQRIXBWBN6XXL7SHLY7W5KC5ES3SFS3A, seg_pos={0 0}, upload=332595]: Long tail cancellation due to optimal threshold Dec 21 10:27:35.055834805 (378h58m34.92691785s) [noise=true , address=77.90.31.15:4700 , piece_id=WQP7T5QHGW23C7I5TBY7PESM5YTTXS3BAEH3RDRQTJZYHI37BUJQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 21 10:27:35.055845194 (378h58m34.926928239s) [noise=true , address=77.90.31.15:4700 , piece_id=WQP7T5QHGW23C7I5TBY7PESM5YTTXS3BAEH3RDRQTJZYHI37BUJQ, seg_pos={0 0}, upload=332595]: Long tail cancellation due to optimal threshold Dec 21 10:27:35.055982826 (378h58m34.927065862s) [noise=true , address=45.91.116.12:4225 , piece_id=WH4P44CPP53IJXXMGK5VK2E3PYBGFLJBWSCFSCBKR2QTALIIEGEA, seg_pos={0 0}, upload=332595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 21 10:27:35.056041539 (378h58m34.927124594s) [noise=true , address=45.91.116.12:4225 , piece_id=WH4P44CPP53IJXXMGK5VK2E3PYBGFLJBWSCFSCBKR2QTALIIEGEA, seg_pos={0 0}, upload=332595]: Long tail cancellation due to optimal threshold Dec 21 10:27:35.056064683 (378h58m34.927147728s) [seg_pos={0 0}, upload=332595]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 10:27:35.056146969 (378h58m34.927230004s) [upload=332595]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 10:27:35.056153121 (378h58m34.927236176s) [upload=332595]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 10:27:35.353226207 (378h58m35.224309242s) [upload=332595]: Done uploading segments... Dec 21 10:27:35.353243942 (378h58m35.224326977s) [upload=332595]: Upload finished. err: <nil> Duration: 5.861909034s core uplink us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 768.34659ms trace: https://jaeger.dev.storj.io/trace/37011901b11802c9 FAIL (2025-12-21 11:00:08 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:37524->45.38.24.2:443: read: connection reset by peer Duration: 768.380747ms edge authservice http us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 5.801232355s trace: https://jaeger.dev.storj.io/trace/3ecb905698746e72 FAIL (2025-12-21 10:59:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: UH4YHD7JFAKXOXJTNKRAUITFZVFWU5AJG5XHSLCJ6JS7JCLWEVRQ) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 3G4VLJ76W4UGBDWZEIQCBDFPRGAM7QL4GMC23LS73BC3JSFGGHPA) context deadline exceeded ======LOGS====== Dec 21 10:27:32.520070050 (378h58m32.391153085s) [upload=332595]: Starting upload Dec 21 10:27:32.520134143 (378h58m32.391217178s) [upload=332595]: Done starting upload Dec 21 10:27:32.520194427 (378h58m32.391277472s) [upload=332595]: Uploading segments... Dec 21 10:27:32.520306451 (378h58m32.391389486s) [upload=332595]: Got next segment. Inline: false Dec 21 10:27:32.520335335 (378h58m32.39141839s) [upload=332595]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 10:27:32.520339694 (378h58m32.391422729s) [upload=332595]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 10:27:32.777396672 (378h58m32.648479718s) [seg_pos={0 0}, upload=332595]: Begin upload segment... Dec 21 10:27:32.777460795 (378h58m32.64854384s) [seg_pos={0 0}, upload=332595]: Done begin upload segment. Dec 21 10:27:32.777515899 (378h58m32.648598934s) [noise=true , address=145.79.227.23:4547 , piece_id=OOD2VMVRTCVGANLHQJME4HCJACBJEH2OUSKZBUY6RVNMSXA3CM7Q, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.777528564 (378h58m32.648611619s) [noise=true , address=145.79.227.27:4691 , piece_id=PEBFRJ6FC4PNFEODQJFGBHHC56RKLUECXWW36YBHNO47HDCDXJWQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.777605250 (378h58m32.648688295s) [noise=true , address=145.79.227.50:4373 , piece_id=GVHEJYGQ6A3HKPZ4V563E37K4L3CZUDEGCVJJHGNC722L3BNY46Q, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.778388482 (378h58m32.649471527s) [noise=true , address=145.79.227.39:4015 , piece_id=QB5S57M5HEYFEZOZYHBWMNFWVLQQEGHS73TBMANOJCCB42LK2AVA, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.778834370 (378h58m32.649917405s) [noise=true , address=45.91.116.12:4225 , piece_id=WH4P44CPP53IJXXMGK5VK2E3PYBGFLJBWSCFSCBKR2QTALIIEGEA, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.778858857 (378h58m32.649941892s) [noise=true , address=145.79.227.42:4121 , piece_id=FA7IZRDCG2XNN6PNDJOTZXGQXZWDOVV2XPR7MUSLK3YVYJVZE4UQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779235654 (378h58m32.650318689s) [noise=true , address=45.91.116.7:4011 , piece_id=UH4YHD7JFAKXOXJTNKRAUITFZVFWU5AJG5XHSLCJ6JS7JCLWEVRQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779269559 (378h58m32.650352594s) [noise=true , address=45.91.116.4:4940 , piece_id=L6QW3XENG6UBHKUUXTMOQQRIXBWBN6XXL7SHLY7W5KC5ES3SFS3A, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779536989 (378h58m32.650620024s) [noise=true , address=145.79.227.52:4445 , piece_id=YWZ6UHBVJWXLIGSE5HMV4WWZLLQLMBCB35WDSUFXK4XMBU5ZAT7Q, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779807954 (378h58m32.650890999s) [noise=true , address=77.90.31.15:4700 , piece_id=WQP7T5QHGW23C7I5TBY7PESM5YTTXS3BAEH3RDRQTJZYHI37BUJQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.780102345 (378h58m32.65118538s) [noise=true , address=145.79.227.17:4972 , piece_id=B3KYU5P3RL2BFR7QTS5ZK7B6UBDB3DTNTQKAGPM7B75S2SYEUFBQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.780482589 (378h58m32.651565634s) [noise=true , address=45.91.116.8:4048 , piece_id=3G4VLJ76W4UGBDWZEIQCBDFPRGAM7QL4GMC23LS73BC3JSFGGHPA, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.780863915 (378h58m32.65194696s) [noise=true , address=145.79.227.48:4303 , piece_id=44PNGUZODJGAD7AFPZGJE6XBDYCSN2CLPIARGZI62PQ7KKU7ANWQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.781783866 (378h58m32.652866911s) [noise=true , address=145.79.227.35:4482 , piece_id=3FUQ4OSF4B5DNWTAPY5W3LS2ZFBLCDWGXXLGASBKACSX3ONKLWWQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.782593317 (378h58m32.653676393s) [noise=true , address=45.91.116.14:4189 , piece_id=NF2SZ7V2LDV76ADGQ6NXVNBHTHNLMRYS4SXW6BT4OM7BQ5TQ6BPQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.783214190 (378h58m32.654297225s) [noise=true , address=145.79.227.51:4411 , piece_id=LC6PCUVNJN77OICI5JGXWN7LJD6JGMKKHHIQG6T4MKX5SU7FGRIQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:34.478418529 (378h58m34.349501564s) [upload=332595]: Next returned nil segment Dec 21 10:27:34.478430381 (378h58m34.349513416s) [upload=332595]: Waiting for error group managing segments... Dec 21 10:27:34.763620982 (378h58m34.634704027s) [noise=true , address=145.79.227.51:4411 , piece_id=LC6PCUVNJN77OICI5JGXWN7LJD6JGMKKHHIQG6T4MKX5SU7FGRIQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.770992478 (378h58m34.642075514s) [noise=true , address=145.79.227.50:4373 , piece_id=GVHEJYGQ6A3HKPZ4V563E37K4L3CZUDEGCVJJHGNC722L3BNY46Q, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.776530073 (378h58m34.647613108s) [noise=true , address=145.79.227.42:4121 , piece_id=FA7IZRDCG2XNN6PNDJOTZXGQXZWDOVV2XPR7MUSLK3YVYJVZE4UQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.786521177 (378h58m34.657604222s) [noise=true , address=145.79.227.17:4972 , piece_id=B3KYU5P3RL2BFR7QTS5ZK7B6UBDB3DTNTQKAGPM7B75S2SYEUFBQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.786591240 (378h58m34.657674275s) [noise=true , address=145.79.227.23:4547 , piece_id=OOD2VMVRTCVGANLHQJME4HCJACBJEH2OUSKZBUY6RVNMSXA3CM7Q, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.787618627 (378h58m34.658701672s) [noise=true , address=145.79.227.52:4445 , piece_id=YWZ6UHBVJWXLIGSE5HMV4WWZLLQLMBCB35WDSUFXK4XMBU5ZAT7Q, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.794650276 (378h58m34.665733311s) [noise=true , address=145.79.227.35:4482 , piece_id=3FUQ4OSF4B5DNWTAPY5W3LS2ZFBLCDWGXXLGASBKACSX3ONKLWWQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.794701083 (378h58m34.665784128s) [noise=true , address=145.79.227.39:4015 , piece_id=QB5S57M5HEYFEZOZYHBWMNFWVLQQEGHS73TBMANOJCCB42LK2AVA, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.833604161 (378h58m34.704687196s) [noise=true , address=145.79.227.48:4303 , piece_id=44PNGUZODJGAD7AFPZGJE6XBDYCSN2CLPIARGZI62PQ7KKU7ANWQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.857890877 (378h58m34.728973912s) [noise=true , address=145.79.227.27:4691 , piece_id=PEBFRJ6FC4PNFEODQJFGBHHC56RKLUECXWW36YBHNO47HDCDXJWQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.991414468 (378h58m34.862497503s) [noise=true , address=45.91.116.7:4011 , piece_id=UH4YHD7JFAKXOXJTNKRAUITFZVFWU5AJG5XHSLCJ6JS7JCLWEVRQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:35.048985192 (378h58m34.920068237s) [noise=true , address=45.91.116.14:4189 , piece_id=NF2SZ7V2LDV76ADGQ6NXVNBHTHNLMRYS4SXW6BT4OM7BQ5TQ6BPQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:35.055578377 (378h58m34.926661412s) [noise=true , address=45.91.116.8:4048 , piece_id=3G4VLJ76W4UGBDWZEIQCBDFPRGAM7QL4GMC23LS73BC3JSFGGHPA, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:35.055594557 (378h58m34.926677602s) [seg_pos={0 0}, upload=332595]: Segment reached optimal threshold of 13 pieces. Dec 21 10:27:35.055773338 (378h58m34.926856373s) [noise=true , address=45.91.116.4:4940 , piece_id=L6QW3XENG6UBHKUUXTMOQQRIXBWBN6XXL7SHLY7W5KC5ES3SFS3A, seg_pos={0 0}, upload=332595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 21 10:27:35.055779740 (378h58m34.926862775s) [noise=true , address=45.91.116.4:4940 , piece_id=L6QW3XENG6UBHKUUXTMOQQRIXBWBN6XXL7SHLY7W5KC5ES3SFS3A, seg_pos={0 0}, upload=332595]: Long tail cancellation due to optimal threshold Dec 21 10:27:35.055834805 (378h58m34.92691785s) [noise=true , address=77.90.31.15:4700 , piece_id=WQP7T5QHGW23C7I5TBY7PESM5YTTXS3BAEH3RDRQTJZYHI37BUJQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 21 10:27:35.055845194 (378h58m34.926928239s) [noise=true , address=77.90.31.15:4700 , piece_id=WQP7T5QHGW23C7I5TBY7PESM5YTTXS3BAEH3RDRQTJZYHI37BUJQ, seg_pos={0 0}, upload=332595]: Long tail cancellation due to optimal threshold Dec 21 10:27:35.055982826 (378h58m34.927065862s) [noise=true , address=45.91.116.12:4225 , piece_id=WH4P44CPP53IJXXMGK5VK2E3PYBGFLJBWSCFSCBKR2QTALIIEGEA, seg_pos={0 0}, upload=332595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 21 10:27:35.056041539 (378h58m34.927124594s) [noise=true , address=45.91.116.12:4225 , piece_id=WH4P44CPP53IJXXMGK5VK2E3PYBGFLJBWSCFSCBKR2QTALIIEGEA, seg_pos={0 0}, upload=332595]: Long tail cancellation due to optimal threshold Dec 21 10:27:35.056064683 (378h58m34.927147728s) [seg_pos={0 0}, upload=332595]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 10:27:35.056146969 (378h58m34.927230004s) [upload=332595]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 10:27:35.056153121 (378h58m34.927236176s) [upload=332595]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 10:27:35.353226207 (378h58m35.224309242s) [upload=332595]: Done uploading segments... Dec 21 10:27:35.353243942 (378h58m35.224326977s) [upload=332595]: Upload finished. err: <nil> Duration: 5.801264416s core uplink us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 758.633436ms trace: https://jaeger.dev.storj.io/trace/37011901b11802c9 FAIL (2025-12-21 10:58:57 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:37524->45.38.24.2:443: read: connection reset by peer Duration: 758.657792ms edge authservice http us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 5.720234694s trace: https://jaeger.dev.storj.io/trace/3ecb905698746e72 FAIL (2025-12-21 10:58:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: UH4YHD7JFAKXOXJTNKRAUITFZVFWU5AJG5XHSLCJ6JS7JCLWEVRQ) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 3G4VLJ76W4UGBDWZEIQCBDFPRGAM7QL4GMC23LS73BC3JSFGGHPA) context deadline exceeded ======LOGS====== Dec 21 10:27:32.520070050 (378h58m32.391153085s) [upload=332595]: Starting upload Dec 21 10:27:32.520134143 (378h58m32.391217178s) [upload=332595]: Done starting upload Dec 21 10:27:32.520194427 (378h58m32.391277472s) [upload=332595]: Uploading segments... Dec 21 10:27:32.520306451 (378h58m32.391389486s) [upload=332595]: Got next segment. Inline: false Dec 21 10:27:32.520335335 (378h58m32.39141839s) [upload=332595]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 10:27:32.520339694 (378h58m32.391422729s) [upload=332595]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 10:27:32.777396672 (378h58m32.648479718s) [seg_pos={0 0}, upload=332595]: Begin upload segment... Dec 21 10:27:32.777460795 (378h58m32.64854384s) [seg_pos={0 0}, upload=332595]: Done begin upload segment. Dec 21 10:27:32.777515899 (378h58m32.648598934s) [noise=true , address=145.79.227.23:4547 , piece_id=OOD2VMVRTCVGANLHQJME4HCJACBJEH2OUSKZBUY6RVNMSXA3CM7Q, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.777528564 (378h58m32.648611619s) [noise=true , address=145.79.227.27:4691 , piece_id=PEBFRJ6FC4PNFEODQJFGBHHC56RKLUECXWW36YBHNO47HDCDXJWQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.777605250 (378h58m32.648688295s) [noise=true , address=145.79.227.50:4373 , piece_id=GVHEJYGQ6A3HKPZ4V563E37K4L3CZUDEGCVJJHGNC722L3BNY46Q, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.778388482 (378h58m32.649471527s) [noise=true , address=145.79.227.39:4015 , piece_id=QB5S57M5HEYFEZOZYHBWMNFWVLQQEGHS73TBMANOJCCB42LK2AVA, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.778834370 (378h58m32.649917405s) [noise=true , address=45.91.116.12:4225 , piece_id=WH4P44CPP53IJXXMGK5VK2E3PYBGFLJBWSCFSCBKR2QTALIIEGEA, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.778858857 (378h58m32.649941892s) [noise=true , address=145.79.227.42:4121 , piece_id=FA7IZRDCG2XNN6PNDJOTZXGQXZWDOVV2XPR7MUSLK3YVYJVZE4UQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779235654 (378h58m32.650318689s) [noise=true , address=45.91.116.7:4011 , piece_id=UH4YHD7JFAKXOXJTNKRAUITFZVFWU5AJG5XHSLCJ6JS7JCLWEVRQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779269559 (378h58m32.650352594s) [noise=true , address=45.91.116.4:4940 , piece_id=L6QW3XENG6UBHKUUXTMOQQRIXBWBN6XXL7SHLY7W5KC5ES3SFS3A, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779536989 (378h58m32.650620024s) [noise=true , address=145.79.227.52:4445 , piece_id=YWZ6UHBVJWXLIGSE5HMV4WWZLLQLMBCB35WDSUFXK4XMBU5ZAT7Q, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779807954 (378h58m32.650890999s) [noise=true , address=77.90.31.15:4700 , piece_id=WQP7T5QHGW23C7I5TBY7PESM5YTTXS3BAEH3RDRQTJZYHI37BUJQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.780102345 (378h58m32.65118538s) [noise=true , address=145.79.227.17:4972 , piece_id=B3KYU5P3RL2BFR7QTS5ZK7B6UBDB3DTNTQKAGPM7B75S2SYEUFBQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.780482589 (378h58m32.651565634s) [noise=true , address=45.91.116.8:4048 , piece_id=3G4VLJ76W4UGBDWZEIQCBDFPRGAM7QL4GMC23LS73BC3JSFGGHPA, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.780863915 (378h58m32.65194696s) [noise=true , address=145.79.227.48:4303 , piece_id=44PNGUZODJGAD7AFPZGJE6XBDYCSN2CLPIARGZI62PQ7KKU7ANWQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.781783866 (378h58m32.652866911s) [noise=true , address=145.79.227.35:4482 , piece_id=3FUQ4OSF4B5DNWTAPY5W3LS2ZFBLCDWGXXLGASBKACSX3ONKLWWQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.782593317 (378h58m32.653676393s) [noise=true , address=45.91.116.14:4189 , piece_id=NF2SZ7V2LDV76ADGQ6NXVNBHTHNLMRYS4SXW6BT4OM7BQ5TQ6BPQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.783214190 (378h58m32.654297225s) [noise=true , address=145.79.227.51:4411 , piece_id=LC6PCUVNJN77OICI5JGXWN7LJD6JGMKKHHIQG6T4MKX5SU7FGRIQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:34.478418529 (378h58m34.349501564s) [upload=332595]: Next returned nil segment Dec 21 10:27:34.478430381 (378h58m34.349513416s) [upload=332595]: Waiting for error group managing segments... Dec 21 10:27:34.763620982 (378h58m34.634704027s) [noise=true , address=145.79.227.51:4411 , piece_id=LC6PCUVNJN77OICI5JGXWN7LJD6JGMKKHHIQG6T4MKX5SU7FGRIQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.770992478 (378h58m34.642075514s) [noise=true , address=145.79.227.50:4373 , piece_id=GVHEJYGQ6A3HKPZ4V563E37K4L3CZUDEGCVJJHGNC722L3BNY46Q, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.776530073 (378h58m34.647613108s) [noise=true , address=145.79.227.42:4121 , piece_id=FA7IZRDCG2XNN6PNDJOTZXGQXZWDOVV2XPR7MUSLK3YVYJVZE4UQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.786521177 (378h58m34.657604222s) [noise=true , address=145.79.227.17:4972 , piece_id=B3KYU5P3RL2BFR7QTS5ZK7B6UBDB3DTNTQKAGPM7B75S2SYEUFBQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.786591240 (378h58m34.657674275s) [noise=true , address=145.79.227.23:4547 , piece_id=OOD2VMVRTCVGANLHQJME4HCJACBJEH2OUSKZBUY6RVNMSXA3CM7Q, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.787618627 (378h58m34.658701672s) [noise=true , address=145.79.227.52:4445 , piece_id=YWZ6UHBVJWXLIGSE5HMV4WWZLLQLMBCB35WDSUFXK4XMBU5ZAT7Q, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.794650276 (378h58m34.665733311s) [noise=true , address=145.79.227.35:4482 , piece_id=3FUQ4OSF4B5DNWTAPY5W3LS2ZFBLCDWGXXLGASBKACSX3ONKLWWQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.794701083 (378h58m34.665784128s) [noise=true , address=145.79.227.39:4015 , piece_id=QB5S57M5HEYFEZOZYHBWMNFWVLQQEGHS73TBMANOJCCB42LK2AVA, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.833604161 (378h58m34.704687196s) [noise=true , address=145.79.227.48:4303 , piece_id=44PNGUZODJGAD7AFPZGJE6XBDYCSN2CLPIARGZI62PQ7KKU7ANWQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.857890877 (378h58m34.728973912s) [noise=true , address=145.79.227.27:4691 , piece_id=PEBFRJ6FC4PNFEODQJFGBHHC56RKLUECXWW36YBHNO47HDCDXJWQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.991414468 (378h58m34.862497503s) [noise=true , address=45.91.116.7:4011 , piece_id=UH4YHD7JFAKXOXJTNKRAUITFZVFWU5AJG5XHSLCJ6JS7JCLWEVRQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:35.048985192 (378h58m34.920068237s) [noise=true , address=45.91.116.14:4189 , piece_id=NF2SZ7V2LDV76ADGQ6NXVNBHTHNLMRYS4SXW6BT4OM7BQ5TQ6BPQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:35.055578377 (378h58m34.926661412s) [noise=true , address=45.91.116.8:4048 , piece_id=3G4VLJ76W4UGBDWZEIQCBDFPRGAM7QL4GMC23LS73BC3JSFGGHPA, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:35.055594557 (378h58m34.926677602s) [seg_pos={0 0}, upload=332595]: Segment reached optimal threshold of 13 pieces. Dec 21 10:27:35.055773338 (378h58m34.926856373s) [noise=true , address=45.91.116.4:4940 , piece_id=L6QW3XENG6UBHKUUXTMOQQRIXBWBN6XXL7SHLY7W5KC5ES3SFS3A, seg_pos={0 0}, upload=332595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 21 10:27:35.055779740 (378h58m34.926862775s) [noise=true , address=45.91.116.4:4940 , piece_id=L6QW3XENG6UBHKUUXTMOQQRIXBWBN6XXL7SHLY7W5KC5ES3SFS3A, seg_pos={0 0}, upload=332595]: Long tail cancellation due to optimal threshold Dec 21 10:27:35.055834805 (378h58m34.92691785s) [noise=true , address=77.90.31.15:4700 , piece_id=WQP7T5QHGW23C7I5TBY7PESM5YTTXS3BAEH3RDRQTJZYHI37BUJQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 21 10:27:35.055845194 (378h58m34.926928239s) [noise=true , address=77.90.31.15:4700 , piece_id=WQP7T5QHGW23C7I5TBY7PESM5YTTXS3BAEH3RDRQTJZYHI37BUJQ, seg_pos={0 0}, upload=332595]: Long tail cancellation due to optimal threshold Dec 21 10:27:35.055982826 (378h58m34.927065862s) [noise=true , address=45.91.116.12:4225 , piece_id=WH4P44CPP53IJXXMGK5VK2E3PYBGFLJBWSCFSCBKR2QTALIIEGEA, seg_pos={0 0}, upload=332595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 21 10:27:35.056041539 (378h58m34.927124594s) [noise=true , address=45.91.116.12:4225 , piece_id=WH4P44CPP53IJXXMGK5VK2E3PYBGFLJBWSCFSCBKR2QTALIIEGEA, seg_pos={0 0}, upload=332595]: Long tail cancellation due to optimal threshold Dec 21 10:27:35.056064683 (378h58m34.927147728s) [seg_pos={0 0}, upload=332595]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 10:27:35.056146969 (378h58m34.927230004s) [upload=332595]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 10:27:35.056153121 (378h58m34.927236176s) [upload=332595]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 10:27:35.353226207 (378h58m35.224309242s) [upload=332595]: Done uploading segments... Dec 21 10:27:35.353243942 (378h58m35.224326977s) [upload=332595]: Upload finished. err: <nil> Duration: 5.720251606s core uplink us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 719.736819ms trace: https://jaeger.dev.storj.io/trace/37011901b11802c9 FAIL (2025-12-21 10:57:59 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:37524->45.38.24.2:443: read: connection reset by peer Duration: 719.761877ms edge authservice http us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 5.904339478s trace: https://jaeger.dev.storj.io/trace/3ecb905698746e72 FAIL (2025-12-21 10:57:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: UH4YHD7JFAKXOXJTNKRAUITFZVFWU5AJG5XHSLCJ6JS7JCLWEVRQ) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 3G4VLJ76W4UGBDWZEIQCBDFPRGAM7QL4GMC23LS73BC3JSFGGHPA) context deadline exceeded ======LOGS====== Dec 21 10:27:32.520070050 (378h58m32.391153085s) [upload=332595]: Starting upload Dec 21 10:27:32.520134143 (378h58m32.391217178s) [upload=332595]: Done starting upload Dec 21 10:27:32.520194427 (378h58m32.391277472s) [upload=332595]: Uploading segments... Dec 21 10:27:32.520306451 (378h58m32.391389486s) [upload=332595]: Got next segment. Inline: false Dec 21 10:27:32.520335335 (378h58m32.39141839s) [upload=332595]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 10:27:32.520339694 (378h58m32.391422729s) [upload=332595]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 10:27:32.777396672 (378h58m32.648479718s) [seg_pos={0 0}, upload=332595]: Begin upload segment... Dec 21 10:27:32.777460795 (378h58m32.64854384s) [seg_pos={0 0}, upload=332595]: Done begin upload segment. Dec 21 10:27:32.777515899 (378h58m32.648598934s) [noise=true , address=145.79.227.23:4547 , piece_id=OOD2VMVRTCVGANLHQJME4HCJACBJEH2OUSKZBUY6RVNMSXA3CM7Q, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.777528564 (378h58m32.648611619s) [noise=true , address=145.79.227.27:4691 , piece_id=PEBFRJ6FC4PNFEODQJFGBHHC56RKLUECXWW36YBHNO47HDCDXJWQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.777605250 (378h58m32.648688295s) [noise=true , address=145.79.227.50:4373 , piece_id=GVHEJYGQ6A3HKPZ4V563E37K4L3CZUDEGCVJJHGNC722L3BNY46Q, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.778388482 (378h58m32.649471527s) [noise=true , address=145.79.227.39:4015 , piece_id=QB5S57M5HEYFEZOZYHBWMNFWVLQQEGHS73TBMANOJCCB42LK2AVA, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.778834370 (378h58m32.649917405s) [noise=true , address=45.91.116.12:4225 , piece_id=WH4P44CPP53IJXXMGK5VK2E3PYBGFLJBWSCFSCBKR2QTALIIEGEA, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.778858857 (378h58m32.649941892s) [noise=true , address=145.79.227.42:4121 , piece_id=FA7IZRDCG2XNN6PNDJOTZXGQXZWDOVV2XPR7MUSLK3YVYJVZE4UQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779235654 (378h58m32.650318689s) [noise=true , address=45.91.116.7:4011 , piece_id=UH4YHD7JFAKXOXJTNKRAUITFZVFWU5AJG5XHSLCJ6JS7JCLWEVRQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779269559 (378h58m32.650352594s) [noise=true , address=45.91.116.4:4940 , piece_id=L6QW3XENG6UBHKUUXTMOQQRIXBWBN6XXL7SHLY7W5KC5ES3SFS3A, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779536989 (378h58m32.650620024s) [noise=true , address=145.79.227.52:4445 , piece_id=YWZ6UHBVJWXLIGSE5HMV4WWZLLQLMBCB35WDSUFXK4XMBU5ZAT7Q, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779807954 (378h58m32.650890999s) [noise=true , address=77.90.31.15:4700 , piece_id=WQP7T5QHGW23C7I5TBY7PESM5YTTXS3BAEH3RDRQTJZYHI37BUJQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.780102345 (378h58m32.65118538s) [noise=true , address=145.79.227.17:4972 , piece_id=B3KYU5P3RL2BFR7QTS5ZK7B6UBDB3DTNTQKAGPM7B75S2SYEUFBQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.780482589 (378h58m32.651565634s) [noise=true , address=45.91.116.8:4048 , piece_id=3G4VLJ76W4UGBDWZEIQCBDFPRGAM7QL4GMC23LS73BC3JSFGGHPA, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.780863915 (378h58m32.65194696s) [noise=true , address=145.79.227.48:4303 , piece_id=44PNGUZODJGAD7AFPZGJE6XBDYCSN2CLPIARGZI62PQ7KKU7ANWQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.781783866 (378h58m32.652866911s) [noise=true , address=145.79.227.35:4482 , piece_id=3FUQ4OSF4B5DNWTAPY5W3LS2ZFBLCDWGXXLGASBKACSX3ONKLWWQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.782593317 (378h58m32.653676393s) [noise=true , address=45.91.116.14:4189 , piece_id=NF2SZ7V2LDV76ADGQ6NXVNBHTHNLMRYS4SXW6BT4OM7BQ5TQ6BPQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.783214190 (378h58m32.654297225s) [noise=true , address=145.79.227.51:4411 , piece_id=LC6PCUVNJN77OICI5JGXWN7LJD6JGMKKHHIQG6T4MKX5SU7FGRIQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:34.478418529 (378h58m34.349501564s) [upload=332595]: Next returned nil segment Dec 21 10:27:34.478430381 (378h58m34.349513416s) [upload=332595]: Waiting for error group managing segments... Dec 21 10:27:34.763620982 (378h58m34.634704027s) [noise=true , address=145.79.227.51:4411 , piece_id=LC6PCUVNJN77OICI5JGXWN7LJD6JGMKKHHIQG6T4MKX5SU7FGRIQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.770992478 (378h58m34.642075514s) [noise=true , address=145.79.227.50:4373 , piece_id=GVHEJYGQ6A3HKPZ4V563E37K4L3CZUDEGCVJJHGNC722L3BNY46Q, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.776530073 (378h58m34.647613108s) [noise=true , address=145.79.227.42:4121 , piece_id=FA7IZRDCG2XNN6PNDJOTZXGQXZWDOVV2XPR7MUSLK3YVYJVZE4UQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.786521177 (378h58m34.657604222s) [noise=true , address=145.79.227.17:4972 , piece_id=B3KYU5P3RL2BFR7QTS5ZK7B6UBDB3DTNTQKAGPM7B75S2SYEUFBQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.786591240 (378h58m34.657674275s) [noise=true , address=145.79.227.23:4547 , piece_id=OOD2VMVRTCVGANLHQJME4HCJACBJEH2OUSKZBUY6RVNMSXA3CM7Q, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.787618627 (378h58m34.658701672s) [noise=true , address=145.79.227.52:4445 , piece_id=YWZ6UHBVJWXLIGSE5HMV4WWZLLQLMBCB35WDSUFXK4XMBU5ZAT7Q, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.794650276 (378h58m34.665733311s) [noise=true , address=145.79.227.35:4482 , piece_id=3FUQ4OSF4B5DNWTAPY5W3LS2ZFBLCDWGXXLGASBKACSX3ONKLWWQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.794701083 (378h58m34.665784128s) [noise=true , address=145.79.227.39:4015 , piece_id=QB5S57M5HEYFEZOZYHBWMNFWVLQQEGHS73TBMANOJCCB42LK2AVA, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.833604161 (378h58m34.704687196s) [noise=true , address=145.79.227.48:4303 , piece_id=44PNGUZODJGAD7AFPZGJE6XBDYCSN2CLPIARGZI62PQ7KKU7ANWQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.857890877 (378h58m34.728973912s) [noise=true , address=145.79.227.27:4691 , piece_id=PEBFRJ6FC4PNFEODQJFGBHHC56RKLUECXWW36YBHNO47HDCDXJWQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.991414468 (378h58m34.862497503s) [noise=true , address=45.91.116.7:4011 , piece_id=UH4YHD7JFAKXOXJTNKRAUITFZVFWU5AJG5XHSLCJ6JS7JCLWEVRQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:35.048985192 (378h58m34.920068237s) [noise=true , address=45.91.116.14:4189 , piece_id=NF2SZ7V2LDV76ADGQ6NXVNBHTHNLMRYS4SXW6BT4OM7BQ5TQ6BPQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:35.055578377 (378h58m34.926661412s) [noise=true , address=45.91.116.8:4048 , piece_id=3G4VLJ76W4UGBDWZEIQCBDFPRGAM7QL4GMC23LS73BC3JSFGGHPA, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:35.055594557 (378h58m34.926677602s) [seg_pos={0 0}, upload=332595]: Segment reached optimal threshold of 13 pieces. Dec 21 10:27:35.055773338 (378h58m34.926856373s) [noise=true , address=45.91.116.4:4940 , piece_id=L6QW3XENG6UBHKUUXTMOQQRIXBWBN6XXL7SHLY7W5KC5ES3SFS3A, seg_pos={0 0}, upload=332595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 21 10:27:35.055779740 (378h58m34.926862775s) [noise=true , address=45.91.116.4:4940 , piece_id=L6QW3XENG6UBHKUUXTMOQQRIXBWBN6XXL7SHLY7W5KC5ES3SFS3A, seg_pos={0 0}, upload=332595]: Long tail cancellation due to optimal threshold Dec 21 10:27:35.055834805 (378h58m34.92691785s) [noise=true , address=77.90.31.15:4700 , piece_id=WQP7T5QHGW23C7I5TBY7PESM5YTTXS3BAEH3RDRQTJZYHI37BUJQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 21 10:27:35.055845194 (378h58m34.926928239s) [noise=true , address=77.90.31.15:4700 , piece_id=WQP7T5QHGW23C7I5TBY7PESM5YTTXS3BAEH3RDRQTJZYHI37BUJQ, seg_pos={0 0}, upload=332595]: Long tail cancellation due to optimal threshold Dec 21 10:27:35.055982826 (378h58m34.927065862s) [noise=true , address=45.91.116.12:4225 , piece_id=WH4P44CPP53IJXXMGK5VK2E3PYBGFLJBWSCFSCBKR2QTALIIEGEA, seg_pos={0 0}, upload=332595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 21 10:27:35.056041539 (378h58m34.927124594s) [noise=true , address=45.91.116.12:4225 , piece_id=WH4P44CPP53IJXXMGK5VK2E3PYBGFLJBWSCFSCBKR2QTALIIEGEA, seg_pos={0 0}, upload=332595]: Long tail cancellation due to optimal threshold Dec 21 10:27:35.056064683 (378h58m34.927147728s) [seg_pos={0 0}, upload=332595]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 10:27:35.056146969 (378h58m34.927230004s) [upload=332595]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 10:27:35.056153121 (378h58m34.927236176s) [upload=332595]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 10:27:35.353226207 (378h58m35.224309242s) [upload=332595]: Done uploading segments... Dec 21 10:27:35.353243942 (378h58m35.224326977s) [upload=332595]: Upload finished. err: <nil> Duration: 5.904366198s core uplink us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 706.226745ms trace: https://jaeger.dev.storj.io/trace/37011901b11802c9 FAIL (2025-12-21 10:57:12 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:37524->45.38.24.2:443: read: connection reset by peer Duration: 706.257854ms edge authservice http us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 5.828271882s trace: https://jaeger.dev.storj.io/trace/3ecb905698746e72 FAIL (2025-12-21 10:56:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: UH4YHD7JFAKXOXJTNKRAUITFZVFWU5AJG5XHSLCJ6JS7JCLWEVRQ) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 3G4VLJ76W4UGBDWZEIQCBDFPRGAM7QL4GMC23LS73BC3JSFGGHPA) context deadline exceeded ======LOGS====== Dec 21 10:27:32.520070050 (378h58m32.391153085s) [upload=332595]: Starting upload Dec 21 10:27:32.520134143 (378h58m32.391217178s) [upload=332595]: Done starting upload Dec 21 10:27:32.520194427 (378h58m32.391277472s) [upload=332595]: Uploading segments... Dec 21 10:27:32.520306451 (378h58m32.391389486s) [upload=332595]: Got next segment. Inline: false Dec 21 10:27:32.520335335 (378h58m32.39141839s) [upload=332595]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 10:27:32.520339694 (378h58m32.391422729s) [upload=332595]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 10:27:32.777396672 (378h58m32.648479718s) [seg_pos={0 0}, upload=332595]: Begin upload segment... Dec 21 10:27:32.777460795 (378h58m32.64854384s) [seg_pos={0 0}, upload=332595]: Done begin upload segment. Dec 21 10:27:32.777515899 (378h58m32.648598934s) [noise=true , address=145.79.227.23:4547 , piece_id=OOD2VMVRTCVGANLHQJME4HCJACBJEH2OUSKZBUY6RVNMSXA3CM7Q, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.777528564 (378h58m32.648611619s) [noise=true , address=145.79.227.27:4691 , piece_id=PEBFRJ6FC4PNFEODQJFGBHHC56RKLUECXWW36YBHNO47HDCDXJWQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.777605250 (378h58m32.648688295s) [noise=true , address=145.79.227.50:4373 , piece_id=GVHEJYGQ6A3HKPZ4V563E37K4L3CZUDEGCVJJHGNC722L3BNY46Q, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.778388482 (378h58m32.649471527s) [noise=true , address=145.79.227.39:4015 , piece_id=QB5S57M5HEYFEZOZYHBWMNFWVLQQEGHS73TBMANOJCCB42LK2AVA, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.778834370 (378h58m32.649917405s) [noise=true , address=45.91.116.12:4225 , piece_id=WH4P44CPP53IJXXMGK5VK2E3PYBGFLJBWSCFSCBKR2QTALIIEGEA, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.778858857 (378h58m32.649941892s) [noise=true , address=145.79.227.42:4121 , piece_id=FA7IZRDCG2XNN6PNDJOTZXGQXZWDOVV2XPR7MUSLK3YVYJVZE4UQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779235654 (378h58m32.650318689s) [noise=true , address=45.91.116.7:4011 , piece_id=UH4YHD7JFAKXOXJTNKRAUITFZVFWU5AJG5XHSLCJ6JS7JCLWEVRQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779269559 (378h58m32.650352594s) [noise=true , address=45.91.116.4:4940 , piece_id=L6QW3XENG6UBHKUUXTMOQQRIXBWBN6XXL7SHLY7W5KC5ES3SFS3A, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779536989 (378h58m32.650620024s) [noise=true , address=145.79.227.52:4445 , piece_id=YWZ6UHBVJWXLIGSE5HMV4WWZLLQLMBCB35WDSUFXK4XMBU5ZAT7Q, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779807954 (378h58m32.650890999s) [noise=true , address=77.90.31.15:4700 , piece_id=WQP7T5QHGW23C7I5TBY7PESM5YTTXS3BAEH3RDRQTJZYHI37BUJQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.780102345 (378h58m32.65118538s) [noise=true , address=145.79.227.17:4972 , piece_id=B3KYU5P3RL2BFR7QTS5ZK7B6UBDB3DTNTQKAGPM7B75S2SYEUFBQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.780482589 (378h58m32.651565634s) [noise=true , address=45.91.116.8:4048 , piece_id=3G4VLJ76W4UGBDWZEIQCBDFPRGAM7QL4GMC23LS73BC3JSFGGHPA, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.780863915 (378h58m32.65194696s) [noise=true , address=145.79.227.48:4303 , piece_id=44PNGUZODJGAD7AFPZGJE6XBDYCSN2CLPIARGZI62PQ7KKU7ANWQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.781783866 (378h58m32.652866911s) [noise=true , address=145.79.227.35:4482 , piece_id=3FUQ4OSF4B5DNWTAPY5W3LS2ZFBLCDWGXXLGASBKACSX3ONKLWWQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.782593317 (378h58m32.653676393s) [noise=true , address=45.91.116.14:4189 , piece_id=NF2SZ7V2LDV76ADGQ6NXVNBHTHNLMRYS4SXW6BT4OM7BQ5TQ6BPQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.783214190 (378h58m32.654297225s) [noise=true , address=145.79.227.51:4411 , piece_id=LC6PCUVNJN77OICI5JGXWN7LJD6JGMKKHHIQG6T4MKX5SU7FGRIQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:34.478418529 (378h58m34.349501564s) [upload=332595]: Next returned nil segment Dec 21 10:27:34.478430381 (378h58m34.349513416s) [upload=332595]: Waiting for error group managing segments... Dec 21 10:27:34.763620982 (378h58m34.634704027s) [noise=true , address=145.79.227.51:4411 , piece_id=LC6PCUVNJN77OICI5JGXWN7LJD6JGMKKHHIQG6T4MKX5SU7FGRIQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.770992478 (378h58m34.642075514s) [noise=true , address=145.79.227.50:4373 , piece_id=GVHEJYGQ6A3HKPZ4V563E37K4L3CZUDEGCVJJHGNC722L3BNY46Q, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.776530073 (378h58m34.647613108s) [noise=true , address=145.79.227.42:4121 , piece_id=FA7IZRDCG2XNN6PNDJOTZXGQXZWDOVV2XPR7MUSLK3YVYJVZE4UQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.786521177 (378h58m34.657604222s) [noise=true , address=145.79.227.17:4972 , piece_id=B3KYU5P3RL2BFR7QTS5ZK7B6UBDB3DTNTQKAGPM7B75S2SYEUFBQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.786591240 (378h58m34.657674275s) [noise=true , address=145.79.227.23:4547 , piece_id=OOD2VMVRTCVGANLHQJME4HCJACBJEH2OUSKZBUY6RVNMSXA3CM7Q, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.787618627 (378h58m34.658701672s) [noise=true , address=145.79.227.52:4445 , piece_id=YWZ6UHBVJWXLIGSE5HMV4WWZLLQLMBCB35WDSUFXK4XMBU5ZAT7Q, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.794650276 (378h58m34.665733311s) [noise=true , address=145.79.227.35:4482 , piece_id=3FUQ4OSF4B5DNWTAPY5W3LS2ZFBLCDWGXXLGASBKACSX3ONKLWWQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.794701083 (378h58m34.665784128s) [noise=true , address=145.79.227.39:4015 , piece_id=QB5S57M5HEYFEZOZYHBWMNFWVLQQEGHS73TBMANOJCCB42LK2AVA, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.833604161 (378h58m34.704687196s) [noise=true , address=145.79.227.48:4303 , piece_id=44PNGUZODJGAD7AFPZGJE6XBDYCSN2CLPIARGZI62PQ7KKU7ANWQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.857890877 (378h58m34.728973912s) [noise=true , address=145.79.227.27:4691 , piece_id=PEBFRJ6FC4PNFEODQJFGBHHC56RKLUECXWW36YBHNO47HDCDXJWQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.991414468 (378h58m34.862497503s) [noise=true , address=45.91.116.7:4011 , piece_id=UH4YHD7JFAKXOXJTNKRAUITFZVFWU5AJG5XHSLCJ6JS7JCLWEVRQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:35.048985192 (378h58m34.920068237s) [noise=true , address=45.91.116.14:4189 , piece_id=NF2SZ7V2LDV76ADGQ6NXVNBHTHNLMRYS4SXW6BT4OM7BQ5TQ6BPQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:35.055578377 (378h58m34.926661412s) [noise=true , address=45.91.116.8:4048 , piece_id=3G4VLJ76W4UGBDWZEIQCBDFPRGAM7QL4GMC23LS73BC3JSFGGHPA, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:35.055594557 (378h58m34.926677602s) [seg_pos={0 0}, upload=332595]: Segment reached optimal threshold of 13 pieces. Dec 21 10:27:35.055773338 (378h58m34.926856373s) [noise=true , address=45.91.116.4:4940 , piece_id=L6QW3XENG6UBHKUUXTMOQQRIXBWBN6XXL7SHLY7W5KC5ES3SFS3A, seg_pos={0 0}, upload=332595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 21 10:27:35.055779740 (378h58m34.926862775s) [noise=true , address=45.91.116.4:4940 , piece_id=L6QW3XENG6UBHKUUXTMOQQRIXBWBN6XXL7SHLY7W5KC5ES3SFS3A, seg_pos={0 0}, upload=332595]: Long tail cancellation due to optimal threshold Dec 21 10:27:35.055834805 (378h58m34.92691785s) [noise=true , address=77.90.31.15:4700 , piece_id=WQP7T5QHGW23C7I5TBY7PESM5YTTXS3BAEH3RDRQTJZYHI37BUJQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 21 10:27:35.055845194 (378h58m34.926928239s) [noise=true , address=77.90.31.15:4700 , piece_id=WQP7T5QHGW23C7I5TBY7PESM5YTTXS3BAEH3RDRQTJZYHI37BUJQ, seg_pos={0 0}, upload=332595]: Long tail cancellation due to optimal threshold Dec 21 10:27:35.055982826 (378h58m34.927065862s) [noise=true , address=45.91.116.12:4225 , piece_id=WH4P44CPP53IJXXMGK5VK2E3PYBGFLJBWSCFSCBKR2QTALIIEGEA, seg_pos={0 0}, upload=332595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 21 10:27:35.056041539 (378h58m34.927124594s) [noise=true , address=45.91.116.12:4225 , piece_id=WH4P44CPP53IJXXMGK5VK2E3PYBGFLJBWSCFSCBKR2QTALIIEGEA, seg_pos={0 0}, upload=332595]: Long tail cancellation due to optimal threshold Dec 21 10:27:35.056064683 (378h58m34.927147728s) [seg_pos={0 0}, upload=332595]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 10:27:35.056146969 (378h58m34.927230004s) [upload=332595]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 10:27:35.056153121 (378h58m34.927236176s) [upload=332595]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 10:27:35.353226207 (378h58m35.224309242s) [upload=332595]: Done uploading segments... Dec 21 10:27:35.353243942 (378h58m35.224326977s) [upload=332595]: Upload finished. err: <nil> Duration: 5.828296759s core uplink us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 791.202211ms trace: https://jaeger.dev.storj.io/trace/37011901b11802c9 FAIL (2025-12-21 10:56:42 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:37524->45.38.24.2:443: read: connection reset by peer Duration: 791.283766ms edge authservice http us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 706.566882ms trace: https://jaeger.dev.storj.io/trace/37011901b11802c9 FAIL (2025-12-21 10:56:10 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:37524->45.38.24.2:443: read: connection reset by peer Duration: 706.595045ms edge authservice http us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 5.761298963s trace: https://jaeger.dev.storj.io/trace/3ecb905698746e72 FAIL (2025-12-21 10:56:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: UH4YHD7JFAKXOXJTNKRAUITFZVFWU5AJG5XHSLCJ6JS7JCLWEVRQ) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 3G4VLJ76W4UGBDWZEIQCBDFPRGAM7QL4GMC23LS73BC3JSFGGHPA) context deadline exceeded ======LOGS====== Dec 21 10:27:32.520070050 (378h58m32.391153085s) [upload=332595]: Starting upload Dec 21 10:27:32.520134143 (378h58m32.391217178s) [upload=332595]: Done starting upload Dec 21 10:27:32.520194427 (378h58m32.391277472s) [upload=332595]: Uploading segments... Dec 21 10:27:32.520306451 (378h58m32.391389486s) [upload=332595]: Got next segment. Inline: false Dec 21 10:27:32.520335335 (378h58m32.39141839s) [upload=332595]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 10:27:32.520339694 (378h58m32.391422729s) [upload=332595]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 10:27:32.777396672 (378h58m32.648479718s) [seg_pos={0 0}, upload=332595]: Begin upload segment... Dec 21 10:27:32.777460795 (378h58m32.64854384s) [seg_pos={0 0}, upload=332595]: Done begin upload segment. Dec 21 10:27:32.777515899 (378h58m32.648598934s) [noise=true , address=145.79.227.23:4547 , piece_id=OOD2VMVRTCVGANLHQJME4HCJACBJEH2OUSKZBUY6RVNMSXA3CM7Q, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.777528564 (378h58m32.648611619s) [noise=true , address=145.79.227.27:4691 , piece_id=PEBFRJ6FC4PNFEODQJFGBHHC56RKLUECXWW36YBHNO47HDCDXJWQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.777605250 (378h58m32.648688295s) [noise=true , address=145.79.227.50:4373 , piece_id=GVHEJYGQ6A3HKPZ4V563E37K4L3CZUDEGCVJJHGNC722L3BNY46Q, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.778388482 (378h58m32.649471527s) [noise=true , address=145.79.227.39:4015 , piece_id=QB5S57M5HEYFEZOZYHBWMNFWVLQQEGHS73TBMANOJCCB42LK2AVA, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.778834370 (378h58m32.649917405s) [noise=true , address=45.91.116.12:4225 , piece_id=WH4P44CPP53IJXXMGK5VK2E3PYBGFLJBWSCFSCBKR2QTALIIEGEA, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.778858857 (378h58m32.649941892s) [noise=true , address=145.79.227.42:4121 , piece_id=FA7IZRDCG2XNN6PNDJOTZXGQXZWDOVV2XPR7MUSLK3YVYJVZE4UQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779235654 (378h58m32.650318689s) [noise=true , address=45.91.116.7:4011 , piece_id=UH4YHD7JFAKXOXJTNKRAUITFZVFWU5AJG5XHSLCJ6JS7JCLWEVRQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779269559 (378h58m32.650352594s) [noise=true , address=45.91.116.4:4940 , piece_id=L6QW3XENG6UBHKUUXTMOQQRIXBWBN6XXL7SHLY7W5KC5ES3SFS3A, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779536989 (378h58m32.650620024s) [noise=true , address=145.79.227.52:4445 , piece_id=YWZ6UHBVJWXLIGSE5HMV4WWZLLQLMBCB35WDSUFXK4XMBU5ZAT7Q, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779807954 (378h58m32.650890999s) [noise=true , address=77.90.31.15:4700 , piece_id=WQP7T5QHGW23C7I5TBY7PESM5YTTXS3BAEH3RDRQTJZYHI37BUJQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.780102345 (378h58m32.65118538s) [noise=true , address=145.79.227.17:4972 , piece_id=B3KYU5P3RL2BFR7QTS5ZK7B6UBDB3DTNTQKAGPM7B75S2SYEUFBQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.780482589 (378h58m32.651565634s) [noise=true , address=45.91.116.8:4048 , piece_id=3G4VLJ76W4UGBDWZEIQCBDFPRGAM7QL4GMC23LS73BC3JSFGGHPA, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.780863915 (378h58m32.65194696s) [noise=true , address=145.79.227.48:4303 , piece_id=44PNGUZODJGAD7AFPZGJE6XBDYCSN2CLPIARGZI62PQ7KKU7ANWQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.781783866 (378h58m32.652866911s) [noise=true , address=145.79.227.35:4482 , piece_id=3FUQ4OSF4B5DNWTAPY5W3LS2ZFBLCDWGXXLGASBKACSX3ONKLWWQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.782593317 (378h58m32.653676393s) [noise=true , address=45.91.116.14:4189 , piece_id=NF2SZ7V2LDV76ADGQ6NXVNBHTHNLMRYS4SXW6BT4OM7BQ5TQ6BPQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.783214190 (378h58m32.654297225s) [noise=true , address=145.79.227.51:4411 , piece_id=LC6PCUVNJN77OICI5JGXWN7LJD6JGMKKHHIQG6T4MKX5SU7FGRIQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:34.478418529 (378h58m34.349501564s) [upload=332595]: Next returned nil segment Dec 21 10:27:34.478430381 (378h58m34.349513416s) [upload=332595]: Waiting for error group managing segments... Dec 21 10:27:34.763620982 (378h58m34.634704027s) [noise=true , address=145.79.227.51:4411 , piece_id=LC6PCUVNJN77OICI5JGXWN7LJD6JGMKKHHIQG6T4MKX5SU7FGRIQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.770992478 (378h58m34.642075514s) [noise=true , address=145.79.227.50:4373 , piece_id=GVHEJYGQ6A3HKPZ4V563E37K4L3CZUDEGCVJJHGNC722L3BNY46Q, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.776530073 (378h58m34.647613108s) [noise=true , address=145.79.227.42:4121 , piece_id=FA7IZRDCG2XNN6PNDJOTZXGQXZWDOVV2XPR7MUSLK3YVYJVZE4UQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.786521177 (378h58m34.657604222s) [noise=true , address=145.79.227.17:4972 , piece_id=B3KYU5P3RL2BFR7QTS5ZK7B6UBDB3DTNTQKAGPM7B75S2SYEUFBQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.786591240 (378h58m34.657674275s) [noise=true , address=145.79.227.23:4547 , piece_id=OOD2VMVRTCVGANLHQJME4HCJACBJEH2OUSKZBUY6RVNMSXA3CM7Q, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.787618627 (378h58m34.658701672s) [noise=true , address=145.79.227.52:4445 , piece_id=YWZ6UHBVJWXLIGSE5HMV4WWZLLQLMBCB35WDSUFXK4XMBU5ZAT7Q, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.794650276 (378h58m34.665733311s) [noise=true , address=145.79.227.35:4482 , piece_id=3FUQ4OSF4B5DNWTAPY5W3LS2ZFBLCDWGXXLGASBKACSX3ONKLWWQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.794701083 (378h58m34.665784128s) [noise=true , address=145.79.227.39:4015 , piece_id=QB5S57M5HEYFEZOZYHBWMNFWVLQQEGHS73TBMANOJCCB42LK2AVA, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.833604161 (378h58m34.704687196s) [noise=true , address=145.79.227.48:4303 , piece_id=44PNGUZODJGAD7AFPZGJE6XBDYCSN2CLPIARGZI62PQ7KKU7ANWQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.857890877 (378h58m34.728973912s) [noise=true , address=145.79.227.27:4691 , piece_id=PEBFRJ6FC4PNFEODQJFGBHHC56RKLUECXWW36YBHNO47HDCDXJWQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.991414468 (378h58m34.862497503s) [noise=true , address=45.91.116.7:4011 , piece_id=UH4YHD7JFAKXOXJTNKRAUITFZVFWU5AJG5XHSLCJ6JS7JCLWEVRQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:35.048985192 (378h58m34.920068237s) [noise=true , address=45.91.116.14:4189 , piece_id=NF2SZ7V2LDV76ADGQ6NXVNBHTHNLMRYS4SXW6BT4OM7BQ5TQ6BPQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:35.055578377 (378h58m34.926661412s) [noise=true , address=45.91.116.8:4048 , piece_id=3G4VLJ76W4UGBDWZEIQCBDFPRGAM7QL4GMC23LS73BC3JSFGGHPA, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:35.055594557 (378h58m34.926677602s) [seg_pos={0 0}, upload=332595]: Segment reached optimal threshold of 13 pieces. Dec 21 10:27:35.055773338 (378h58m34.926856373s) [noise=true , address=45.91.116.4:4940 , piece_id=L6QW3XENG6UBHKUUXTMOQQRIXBWBN6XXL7SHLY7W5KC5ES3SFS3A, seg_pos={0 0}, upload=332595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 21 10:27:35.055779740 (378h58m34.926862775s) [noise=true , address=45.91.116.4:4940 , piece_id=L6QW3XENG6UBHKUUXTMOQQRIXBWBN6XXL7SHLY7W5KC5ES3SFS3A, seg_pos={0 0}, upload=332595]: Long tail cancellation due to optimal threshold Dec 21 10:27:35.055834805 (378h58m34.92691785s) [noise=true , address=77.90.31.15:4700 , piece_id=WQP7T5QHGW23C7I5TBY7PESM5YTTXS3BAEH3RDRQTJZYHI37BUJQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 21 10:27:35.055845194 (378h58m34.926928239s) [noise=true , address=77.90.31.15:4700 , piece_id=WQP7T5QHGW23C7I5TBY7PESM5YTTXS3BAEH3RDRQTJZYHI37BUJQ, seg_pos={0 0}, upload=332595]: Long tail cancellation due to optimal threshold Dec 21 10:27:35.055982826 (378h58m34.927065862s) [noise=true , address=45.91.116.12:4225 , piece_id=WH4P44CPP53IJXXMGK5VK2E3PYBGFLJBWSCFSCBKR2QTALIIEGEA, seg_pos={0 0}, upload=332595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 21 10:27:35.056041539 (378h58m34.927124594s) [noise=true , address=45.91.116.12:4225 , piece_id=WH4P44CPP53IJXXMGK5VK2E3PYBGFLJBWSCFSCBKR2QTALIIEGEA, seg_pos={0 0}, upload=332595]: Long tail cancellation due to optimal threshold Dec 21 10:27:35.056064683 (378h58m34.927147728s) [seg_pos={0 0}, upload=332595]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 10:27:35.056146969 (378h58m34.927230004s) [upload=332595]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 10:27:35.056153121 (378h58m34.927236176s) [upload=332595]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 10:27:35.353226207 (378h58m35.224309242s) [upload=332595]: Done uploading segments... Dec 21 10:27:35.353243942 (378h58m35.224326977s) [upload=332595]: Upload finished. err: <nil> Duration: 5.761324943s core uplink us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 689.595878ms trace: https://jaeger.dev.storj.io/trace/37011901b11802c9 FAIL (2025-12-21 10:55:10 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:37524->45.38.24.2:443: read: connection reset by peer Duration: 689.611809ms edge authservice http us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 5.695582068s trace: https://jaeger.dev.storj.io/trace/3ecb905698746e72 FAIL (2025-12-21 10:54:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: UH4YHD7JFAKXOXJTNKRAUITFZVFWU5AJG5XHSLCJ6JS7JCLWEVRQ) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 3G4VLJ76W4UGBDWZEIQCBDFPRGAM7QL4GMC23LS73BC3JSFGGHPA) context deadline exceeded ======LOGS====== Dec 21 10:27:32.520070050 (378h58m32.391153085s) [upload=332595]: Starting upload Dec 21 10:27:32.520134143 (378h58m32.391217178s) [upload=332595]: Done starting upload Dec 21 10:27:32.520194427 (378h58m32.391277472s) [upload=332595]: Uploading segments... Dec 21 10:27:32.520306451 (378h58m32.391389486s) [upload=332595]: Got next segment. Inline: false Dec 21 10:27:32.520335335 (378h58m32.39141839s) [upload=332595]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 10:27:32.520339694 (378h58m32.391422729s) [upload=332595]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 10:27:32.777396672 (378h58m32.648479718s) [seg_pos={0 0}, upload=332595]: Begin upload segment... Dec 21 10:27:32.777460795 (378h58m32.64854384s) [seg_pos={0 0}, upload=332595]: Done begin upload segment. Dec 21 10:27:32.777515899 (378h58m32.648598934s) [noise=true , address=145.79.227.23:4547 , piece_id=OOD2VMVRTCVGANLHQJME4HCJACBJEH2OUSKZBUY6RVNMSXA3CM7Q, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.777528564 (378h58m32.648611619s) [noise=true , address=145.79.227.27:4691 , piece_id=PEBFRJ6FC4PNFEODQJFGBHHC56RKLUECXWW36YBHNO47HDCDXJWQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.777605250 (378h58m32.648688295s) [noise=true , address=145.79.227.50:4373 , piece_id=GVHEJYGQ6A3HKPZ4V563E37K4L3CZUDEGCVJJHGNC722L3BNY46Q, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.778388482 (378h58m32.649471527s) [noise=true , address=145.79.227.39:4015 , piece_id=QB5S57M5HEYFEZOZYHBWMNFWVLQQEGHS73TBMANOJCCB42LK2AVA, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.778834370 (378h58m32.649917405s) [noise=true , address=45.91.116.12:4225 , piece_id=WH4P44CPP53IJXXMGK5VK2E3PYBGFLJBWSCFSCBKR2QTALIIEGEA, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.778858857 (378h58m32.649941892s) [noise=true , address=145.79.227.42:4121 , piece_id=FA7IZRDCG2XNN6PNDJOTZXGQXZWDOVV2XPR7MUSLK3YVYJVZE4UQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779235654 (378h58m32.650318689s) [noise=true , address=45.91.116.7:4011 , piece_id=UH4YHD7JFAKXOXJTNKRAUITFZVFWU5AJG5XHSLCJ6JS7JCLWEVRQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779269559 (378h58m32.650352594s) [noise=true , address=45.91.116.4:4940 , piece_id=L6QW3XENG6UBHKUUXTMOQQRIXBWBN6XXL7SHLY7W5KC5ES3SFS3A, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779536989 (378h58m32.650620024s) [noise=true , address=145.79.227.52:4445 , piece_id=YWZ6UHBVJWXLIGSE5HMV4WWZLLQLMBCB35WDSUFXK4XMBU5ZAT7Q, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779807954 (378h58m32.650890999s) [noise=true , address=77.90.31.15:4700 , piece_id=WQP7T5QHGW23C7I5TBY7PESM5YTTXS3BAEH3RDRQTJZYHI37BUJQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.780102345 (378h58m32.65118538s) [noise=true , address=145.79.227.17:4972 , piece_id=B3KYU5P3RL2BFR7QTS5ZK7B6UBDB3DTNTQKAGPM7B75S2SYEUFBQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.780482589 (378h58m32.651565634s) [noise=true , address=45.91.116.8:4048 , piece_id=3G4VLJ76W4UGBDWZEIQCBDFPRGAM7QL4GMC23LS73BC3JSFGGHPA, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.780863915 (378h58m32.65194696s) [noise=true , address=145.79.227.48:4303 , piece_id=44PNGUZODJGAD7AFPZGJE6XBDYCSN2CLPIARGZI62PQ7KKU7ANWQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.781783866 (378h58m32.652866911s) [noise=true , address=145.79.227.35:4482 , piece_id=3FUQ4OSF4B5DNWTAPY5W3LS2ZFBLCDWGXXLGASBKACSX3ONKLWWQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.782593317 (378h58m32.653676393s) [noise=true , address=45.91.116.14:4189 , piece_id=NF2SZ7V2LDV76ADGQ6NXVNBHTHNLMRYS4SXW6BT4OM7BQ5TQ6BPQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.783214190 (378h58m32.654297225s) [noise=true , address=145.79.227.51:4411 , piece_id=LC6PCUVNJN77OICI5JGXWN7LJD6JGMKKHHIQG6T4MKX5SU7FGRIQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:34.478418529 (378h58m34.349501564s) [upload=332595]: Next returned nil segment Dec 21 10:27:34.478430381 (378h58m34.349513416s) [upload=332595]: Waiting for error group managing segments... Dec 21 10:27:34.763620982 (378h58m34.634704027s) [noise=true , address=145.79.227.51:4411 , piece_id=LC6PCUVNJN77OICI5JGXWN7LJD6JGMKKHHIQG6T4MKX5SU7FGRIQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.770992478 (378h58m34.642075514s) [noise=true , address=145.79.227.50:4373 , piece_id=GVHEJYGQ6A3HKPZ4V563E37K4L3CZUDEGCVJJHGNC722L3BNY46Q, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.776530073 (378h58m34.647613108s) [noise=true , address=145.79.227.42:4121 , piece_id=FA7IZRDCG2XNN6PNDJOTZXGQXZWDOVV2XPR7MUSLK3YVYJVZE4UQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.786521177 (378h58m34.657604222s) [noise=true , address=145.79.227.17:4972 , piece_id=B3KYU5P3RL2BFR7QTS5ZK7B6UBDB3DTNTQKAGPM7B75S2SYEUFBQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.786591240 (378h58m34.657674275s) [noise=true , address=145.79.227.23:4547 , piece_id=OOD2VMVRTCVGANLHQJME4HCJACBJEH2OUSKZBUY6RVNMSXA3CM7Q, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.787618627 (378h58m34.658701672s) [noise=true , address=145.79.227.52:4445 , piece_id=YWZ6UHBVJWXLIGSE5HMV4WWZLLQLMBCB35WDSUFXK4XMBU5ZAT7Q, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.794650276 (378h58m34.665733311s) [noise=true , address=145.79.227.35:4482 , piece_id=3FUQ4OSF4B5DNWTAPY5W3LS2ZFBLCDWGXXLGASBKACSX3ONKLWWQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.794701083 (378h58m34.665784128s) [noise=true , address=145.79.227.39:4015 , piece_id=QB5S57M5HEYFEZOZYHBWMNFWVLQQEGHS73TBMANOJCCB42LK2AVA, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.833604161 (378h58m34.704687196s) [noise=true , address=145.79.227.48:4303 , piece_id=44PNGUZODJGAD7AFPZGJE6XBDYCSN2CLPIARGZI62PQ7KKU7ANWQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.857890877 (378h58m34.728973912s) [noise=true , address=145.79.227.27:4691 , piece_id=PEBFRJ6FC4PNFEODQJFGBHHC56RKLUECXWW36YBHNO47HDCDXJWQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.991414468 (378h58m34.862497503s) [noise=true , address=45.91.116.7:4011 , piece_id=UH4YHD7JFAKXOXJTNKRAUITFZVFWU5AJG5XHSLCJ6JS7JCLWEVRQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:35.048985192 (378h58m34.920068237s) [noise=true , address=45.91.116.14:4189 , piece_id=NF2SZ7V2LDV76ADGQ6NXVNBHTHNLMRYS4SXW6BT4OM7BQ5TQ6BPQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:35.055578377 (378h58m34.926661412s) [noise=true , address=45.91.116.8:4048 , piece_id=3G4VLJ76W4UGBDWZEIQCBDFPRGAM7QL4GMC23LS73BC3JSFGGHPA, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:35.055594557 (378h58m34.926677602s) [seg_pos={0 0}, upload=332595]: Segment reached optimal threshold of 13 pieces. Dec 21 10:27:35.055773338 (378h58m34.926856373s) [noise=true , address=45.91.116.4:4940 , piece_id=L6QW3XENG6UBHKUUXTMOQQRIXBWBN6XXL7SHLY7W5KC5ES3SFS3A, seg_pos={0 0}, upload=332595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 21 10:27:35.055779740 (378h58m34.926862775s) [noise=true , address=45.91.116.4:4940 , piece_id=L6QW3XENG6UBHKUUXTMOQQRIXBWBN6XXL7SHLY7W5KC5ES3SFS3A, seg_pos={0 0}, upload=332595]: Long tail cancellation due to optimal threshold Dec 21 10:27:35.055834805 (378h58m34.92691785s) [noise=true , address=77.90.31.15:4700 , piece_id=WQP7T5QHGW23C7I5TBY7PESM5YTTXS3BAEH3RDRQTJZYHI37BUJQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 21 10:27:35.055845194 (378h58m34.926928239s) [noise=true , address=77.90.31.15:4700 , piece_id=WQP7T5QHGW23C7I5TBY7PESM5YTTXS3BAEH3RDRQTJZYHI37BUJQ, seg_pos={0 0}, upload=332595]: Long tail cancellation due to optimal threshold Dec 21 10:27:35.055982826 (378h58m34.927065862s) [noise=true , address=45.91.116.12:4225 , piece_id=WH4P44CPP53IJXXMGK5VK2E3PYBGFLJBWSCFSCBKR2QTALIIEGEA, seg_pos={0 0}, upload=332595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 21 10:27:35.056041539 (378h58m34.927124594s) [noise=true , address=45.91.116.12:4225 , piece_id=WH4P44CPP53IJXXMGK5VK2E3PYBGFLJBWSCFSCBKR2QTALIIEGEA, seg_pos={0 0}, upload=332595]: Long tail cancellation due to optimal threshold Dec 21 10:27:35.056064683 (378h58m34.927147728s) [seg_pos={0 0}, upload=332595]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 10:27:35.056146969 (378h58m34.927230004s) [upload=332595]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 10:27:35.056153121 (378h58m34.927236176s) [upload=332595]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 10:27:35.353226207 (378h58m35.224309242s) [upload=332595]: Done uploading segments... Dec 21 10:27:35.353243942 (378h58m35.224326977s) [upload=332595]: Upload finished. err: <nil> Duration: 5.695623517s core uplink us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 721.320514ms trace: https://jaeger.dev.storj.io/trace/37011901b11802c9 FAIL (2025-12-21 10:54:30 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:37524->45.38.24.2:443: read: connection reset by peer Duration: 721.353256ms edge authservice http us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 6.488829417s trace: https://jaeger.dev.storj.io/trace/3ecb905698746e72 FAIL (2025-12-21 10:53:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: UH4YHD7JFAKXOXJTNKRAUITFZVFWU5AJG5XHSLCJ6JS7JCLWEVRQ) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 3G4VLJ76W4UGBDWZEIQCBDFPRGAM7QL4GMC23LS73BC3JSFGGHPA) context deadline exceeded ======LOGS====== Dec 21 10:27:32.520070050 (378h58m32.391153085s) [upload=332595]: Starting upload Dec 21 10:27:32.520134143 (378h58m32.391217178s) [upload=332595]: Done starting upload Dec 21 10:27:32.520194427 (378h58m32.391277472s) [upload=332595]: Uploading segments... Dec 21 10:27:32.520306451 (378h58m32.391389486s) [upload=332595]: Got next segment. Inline: false Dec 21 10:27:32.520335335 (378h58m32.39141839s) [upload=332595]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 10:27:32.520339694 (378h58m32.391422729s) [upload=332595]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 10:27:32.777396672 (378h58m32.648479718s) [seg_pos={0 0}, upload=332595]: Begin upload segment... Dec 21 10:27:32.777460795 (378h58m32.64854384s) [seg_pos={0 0}, upload=332595]: Done begin upload segment. Dec 21 10:27:32.777515899 (378h58m32.648598934s) [noise=true , address=145.79.227.23:4547 , piece_id=OOD2VMVRTCVGANLHQJME4HCJACBJEH2OUSKZBUY6RVNMSXA3CM7Q, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.777528564 (378h58m32.648611619s) [noise=true , address=145.79.227.27:4691 , piece_id=PEBFRJ6FC4PNFEODQJFGBHHC56RKLUECXWW36YBHNO47HDCDXJWQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.777605250 (378h58m32.648688295s) [noise=true , address=145.79.227.50:4373 , piece_id=GVHEJYGQ6A3HKPZ4V563E37K4L3CZUDEGCVJJHGNC722L3BNY46Q, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.778388482 (378h58m32.649471527s) [noise=true , address=145.79.227.39:4015 , piece_id=QB5S57M5HEYFEZOZYHBWMNFWVLQQEGHS73TBMANOJCCB42LK2AVA, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.778834370 (378h58m32.649917405s) [noise=true , address=45.91.116.12:4225 , piece_id=WH4P44CPP53IJXXMGK5VK2E3PYBGFLJBWSCFSCBKR2QTALIIEGEA, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.778858857 (378h58m32.649941892s) [noise=true , address=145.79.227.42:4121 , piece_id=FA7IZRDCG2XNN6PNDJOTZXGQXZWDOVV2XPR7MUSLK3YVYJVZE4UQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779235654 (378h58m32.650318689s) [noise=true , address=45.91.116.7:4011 , piece_id=UH4YHD7JFAKXOXJTNKRAUITFZVFWU5AJG5XHSLCJ6JS7JCLWEVRQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779269559 (378h58m32.650352594s) [noise=true , address=45.91.116.4:4940 , piece_id=L6QW3XENG6UBHKUUXTMOQQRIXBWBN6XXL7SHLY7W5KC5ES3SFS3A, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779536989 (378h58m32.650620024s) [noise=true , address=145.79.227.52:4445 , piece_id=YWZ6UHBVJWXLIGSE5HMV4WWZLLQLMBCB35WDSUFXK4XMBU5ZAT7Q, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779807954 (378h58m32.650890999s) [noise=true , address=77.90.31.15:4700 , piece_id=WQP7T5QHGW23C7I5TBY7PESM5YTTXS3BAEH3RDRQTJZYHI37BUJQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.780102345 (378h58m32.65118538s) [noise=true , address=145.79.227.17:4972 , piece_id=B3KYU5P3RL2BFR7QTS5ZK7B6UBDB3DTNTQKAGPM7B75S2SYEUFBQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.780482589 (378h58m32.651565634s) [noise=true , address=45.91.116.8:4048 , piece_id=3G4VLJ76W4UGBDWZEIQCBDFPRGAM7QL4GMC23LS73BC3JSFGGHPA, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.780863915 (378h58m32.65194696s) [noise=true , address=145.79.227.48:4303 , piece_id=44PNGUZODJGAD7AFPZGJE6XBDYCSN2CLPIARGZI62PQ7KKU7ANWQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.781783866 (378h58m32.652866911s) [noise=true , address=145.79.227.35:4482 , piece_id=3FUQ4OSF4B5DNWTAPY5W3LS2ZFBLCDWGXXLGASBKACSX3ONKLWWQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.782593317 (378h58m32.653676393s) [noise=true , address=45.91.116.14:4189 , piece_id=NF2SZ7V2LDV76ADGQ6NXVNBHTHNLMRYS4SXW6BT4OM7BQ5TQ6BPQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.783214190 (378h58m32.654297225s) [noise=true , address=145.79.227.51:4411 , piece_id=LC6PCUVNJN77OICI5JGXWN7LJD6JGMKKHHIQG6T4MKX5SU7FGRIQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:34.478418529 (378h58m34.349501564s) [upload=332595]: Next returned nil segment Dec 21 10:27:34.478430381 (378h58m34.349513416s) [upload=332595]: Waiting for error group managing segments... Dec 21 10:27:34.763620982 (378h58m34.634704027s) [noise=true , address=145.79.227.51:4411 , piece_id=LC6PCUVNJN77OICI5JGXWN7LJD6JGMKKHHIQG6T4MKX5SU7FGRIQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.770992478 (378h58m34.642075514s) [noise=true , address=145.79.227.50:4373 , piece_id=GVHEJYGQ6A3HKPZ4V563E37K4L3CZUDEGCVJJHGNC722L3BNY46Q, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.776530073 (378h58m34.647613108s) [noise=true , address=145.79.227.42:4121 , piece_id=FA7IZRDCG2XNN6PNDJOTZXGQXZWDOVV2XPR7MUSLK3YVYJVZE4UQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.786521177 (378h58m34.657604222s) [noise=true , address=145.79.227.17:4972 , piece_id=B3KYU5P3RL2BFR7QTS5ZK7B6UBDB3DTNTQKAGPM7B75S2SYEUFBQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.786591240 (378h58m34.657674275s) [noise=true , address=145.79.227.23:4547 , piece_id=OOD2VMVRTCVGANLHQJME4HCJACBJEH2OUSKZBUY6RVNMSXA3CM7Q, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.787618627 (378h58m34.658701672s) [noise=true , address=145.79.227.52:4445 , piece_id=YWZ6UHBVJWXLIGSE5HMV4WWZLLQLMBCB35WDSUFXK4XMBU5ZAT7Q, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.794650276 (378h58m34.665733311s) [noise=true , address=145.79.227.35:4482 , piece_id=3FUQ4OSF4B5DNWTAPY5W3LS2ZFBLCDWGXXLGASBKACSX3ONKLWWQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.794701083 (378h58m34.665784128s) [noise=true , address=145.79.227.39:4015 , piece_id=QB5S57M5HEYFEZOZYHBWMNFWVLQQEGHS73TBMANOJCCB42LK2AVA, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.833604161 (378h58m34.704687196s) [noise=true , address=145.79.227.48:4303 , piece_id=44PNGUZODJGAD7AFPZGJE6XBDYCSN2CLPIARGZI62PQ7KKU7ANWQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.857890877 (378h58m34.728973912s) [noise=true , address=145.79.227.27:4691 , piece_id=PEBFRJ6FC4PNFEODQJFGBHHC56RKLUECXWW36YBHNO47HDCDXJWQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.991414468 (378h58m34.862497503s) [noise=true , address=45.91.116.7:4011 , piece_id=UH4YHD7JFAKXOXJTNKRAUITFZVFWU5AJG5XHSLCJ6JS7JCLWEVRQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:35.048985192 (378h58m34.920068237s) [noise=true , address=45.91.116.14:4189 , piece_id=NF2SZ7V2LDV76ADGQ6NXVNBHTHNLMRYS4SXW6BT4OM7BQ5TQ6BPQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:35.055578377 (378h58m34.926661412s) [noise=true , address=45.91.116.8:4048 , piece_id=3G4VLJ76W4UGBDWZEIQCBDFPRGAM7QL4GMC23LS73BC3JSFGGHPA, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:35.055594557 (378h58m34.926677602s) [seg_pos={0 0}, upload=332595]: Segment reached optimal threshold of 13 pieces. Dec 21 10:27:35.055773338 (378h58m34.926856373s) [noise=true , address=45.91.116.4:4940 , piece_id=L6QW3XENG6UBHKUUXTMOQQRIXBWBN6XXL7SHLY7W5KC5ES3SFS3A, seg_pos={0 0}, upload=332595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 21 10:27:35.055779740 (378h58m34.926862775s) [noise=true , address=45.91.116.4:4940 , piece_id=L6QW3XENG6UBHKUUXTMOQQRIXBWBN6XXL7SHLY7W5KC5ES3SFS3A, seg_pos={0 0}, upload=332595]: Long tail cancellation due to optimal threshold Dec 21 10:27:35.055834805 (378h58m34.92691785s) [noise=true , address=77.90.31.15:4700 , piece_id=WQP7T5QHGW23C7I5TBY7PESM5YTTXS3BAEH3RDRQTJZYHI37BUJQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 21 10:27:35.055845194 (378h58m34.926928239s) [noise=true , address=77.90.31.15:4700 , piece_id=WQP7T5QHGW23C7I5TBY7PESM5YTTXS3BAEH3RDRQTJZYHI37BUJQ, seg_pos={0 0}, upload=332595]: Long tail cancellation due to optimal threshold Dec 21 10:27:35.055982826 (378h58m34.927065862s) [noise=true , address=45.91.116.12:4225 , piece_id=WH4P44CPP53IJXXMGK5VK2E3PYBGFLJBWSCFSCBKR2QTALIIEGEA, seg_pos={0 0}, upload=332595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 21 10:27:35.056041539 (378h58m34.927124594s) [noise=true , address=45.91.116.12:4225 , piece_id=WH4P44CPP53IJXXMGK5VK2E3PYBGFLJBWSCFSCBKR2QTALIIEGEA, seg_pos={0 0}, upload=332595]: Long tail cancellation due to optimal threshold Dec 21 10:27:35.056064683 (378h58m34.927147728s) [seg_pos={0 0}, upload=332595]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 10:27:35.056146969 (378h58m34.927230004s) [upload=332595]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 10:27:35.056153121 (378h58m34.927236176s) [upload=332595]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 10:27:35.353226207 (378h58m35.224309242s) [upload=332595]: Done uploading segments... Dec 21 10:27:35.353243942 (378h58m35.224326977s) [upload=332595]: Upload finished. err: <nil> Duration: 6.488859034s core uplink us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 745.633132ms trace: https://jaeger.dev.storj.io/trace/37011901b11802c9 FAIL (2025-12-21 10:53:45 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:37524->45.38.24.2:443: read: connection reset by peer Duration: 745.659802ms edge authservice http us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 756.745981ms trace: https://jaeger.dev.storj.io/trace/37011901b11802c9 FAIL (2025-12-21 10:53:19 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:37524->45.38.24.2:443: read: connection reset by peer Duration: 756.823709ms edge authservice http us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 5.685043043s trace: https://jaeger.dev.storj.io/trace/3ecb905698746e72 FAIL (2025-12-21 10:53:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: UH4YHD7JFAKXOXJTNKRAUITFZVFWU5AJG5XHSLCJ6JS7JCLWEVRQ) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 3G4VLJ76W4UGBDWZEIQCBDFPRGAM7QL4GMC23LS73BC3JSFGGHPA) context deadline exceeded ======LOGS====== Dec 21 10:27:32.520070050 (378h58m32.391153085s) [upload=332595]: Starting upload Dec 21 10:27:32.520134143 (378h58m32.391217178s) [upload=332595]: Done starting upload Dec 21 10:27:32.520194427 (378h58m32.391277472s) [upload=332595]: Uploading segments... Dec 21 10:27:32.520306451 (378h58m32.391389486s) [upload=332595]: Got next segment. Inline: false Dec 21 10:27:32.520335335 (378h58m32.39141839s) [upload=332595]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 10:27:32.520339694 (378h58m32.391422729s) [upload=332595]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 10:27:32.777396672 (378h58m32.648479718s) [seg_pos={0 0}, upload=332595]: Begin upload segment... Dec 21 10:27:32.777460795 (378h58m32.64854384s) [seg_pos={0 0}, upload=332595]: Done begin upload segment. Dec 21 10:27:32.777515899 (378h58m32.648598934s) [noise=true , address=145.79.227.23:4547 , piece_id=OOD2VMVRTCVGANLHQJME4HCJACBJEH2OUSKZBUY6RVNMSXA3CM7Q, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.777528564 (378h58m32.648611619s) [noise=true , address=145.79.227.27:4691 , piece_id=PEBFRJ6FC4PNFEODQJFGBHHC56RKLUECXWW36YBHNO47HDCDXJWQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.777605250 (378h58m32.648688295s) [noise=true , address=145.79.227.50:4373 , piece_id=GVHEJYGQ6A3HKPZ4V563E37K4L3CZUDEGCVJJHGNC722L3BNY46Q, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.778388482 (378h58m32.649471527s) [noise=true , address=145.79.227.39:4015 , piece_id=QB5S57M5HEYFEZOZYHBWMNFWVLQQEGHS73TBMANOJCCB42LK2AVA, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.778834370 (378h58m32.649917405s) [noise=true , address=45.91.116.12:4225 , piece_id=WH4P44CPP53IJXXMGK5VK2E3PYBGFLJBWSCFSCBKR2QTALIIEGEA, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.778858857 (378h58m32.649941892s) [noise=true , address=145.79.227.42:4121 , piece_id=FA7IZRDCG2XNN6PNDJOTZXGQXZWDOVV2XPR7MUSLK3YVYJVZE4UQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779235654 (378h58m32.650318689s) [noise=true , address=45.91.116.7:4011 , piece_id=UH4YHD7JFAKXOXJTNKRAUITFZVFWU5AJG5XHSLCJ6JS7JCLWEVRQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779269559 (378h58m32.650352594s) [noise=true , address=45.91.116.4:4940 , piece_id=L6QW3XENG6UBHKUUXTMOQQRIXBWBN6XXL7SHLY7W5KC5ES3SFS3A, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779536989 (378h58m32.650620024s) [noise=true , address=145.79.227.52:4445 , piece_id=YWZ6UHBVJWXLIGSE5HMV4WWZLLQLMBCB35WDSUFXK4XMBU5ZAT7Q, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779807954 (378h58m32.650890999s) [noise=true , address=77.90.31.15:4700 , piece_id=WQP7T5QHGW23C7I5TBY7PESM5YTTXS3BAEH3RDRQTJZYHI37BUJQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.780102345 (378h58m32.65118538s) [noise=true , address=145.79.227.17:4972 , piece_id=B3KYU5P3RL2BFR7QTS5ZK7B6UBDB3DTNTQKAGPM7B75S2SYEUFBQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.780482589 (378h58m32.651565634s) [noise=true , address=45.91.116.8:4048 , piece_id=3G4VLJ76W4UGBDWZEIQCBDFPRGAM7QL4GMC23LS73BC3JSFGGHPA, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.780863915 (378h58m32.65194696s) [noise=true , address=145.79.227.48:4303 , piece_id=44PNGUZODJGAD7AFPZGJE6XBDYCSN2CLPIARGZI62PQ7KKU7ANWQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.781783866 (378h58m32.652866911s) [noise=true , address=145.79.227.35:4482 , piece_id=3FUQ4OSF4B5DNWTAPY5W3LS2ZFBLCDWGXXLGASBKACSX3ONKLWWQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.782593317 (378h58m32.653676393s) [noise=true , address=45.91.116.14:4189 , piece_id=NF2SZ7V2LDV76ADGQ6NXVNBHTHNLMRYS4SXW6BT4OM7BQ5TQ6BPQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.783214190 (378h58m32.654297225s) [noise=true , address=145.79.227.51:4411 , piece_id=LC6PCUVNJN77OICI5JGXWN7LJD6JGMKKHHIQG6T4MKX5SU7FGRIQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:34.478418529 (378h58m34.349501564s) [upload=332595]: Next returned nil segment Dec 21 10:27:34.478430381 (378h58m34.349513416s) [upload=332595]: Waiting for error group managing segments... Dec 21 10:27:34.763620982 (378h58m34.634704027s) [noise=true , address=145.79.227.51:4411 , piece_id=LC6PCUVNJN77OICI5JGXWN7LJD6JGMKKHHIQG6T4MKX5SU7FGRIQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.770992478 (378h58m34.642075514s) [noise=true , address=145.79.227.50:4373 , piece_id=GVHEJYGQ6A3HKPZ4V563E37K4L3CZUDEGCVJJHGNC722L3BNY46Q, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.776530073 (378h58m34.647613108s) [noise=true , address=145.79.227.42:4121 , piece_id=FA7IZRDCG2XNN6PNDJOTZXGQXZWDOVV2XPR7MUSLK3YVYJVZE4UQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.786521177 (378h58m34.657604222s) [noise=true , address=145.79.227.17:4972 , piece_id=B3KYU5P3RL2BFR7QTS5ZK7B6UBDB3DTNTQKAGPM7B75S2SYEUFBQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.786591240 (378h58m34.657674275s) [noise=true , address=145.79.227.23:4547 , piece_id=OOD2VMVRTCVGANLHQJME4HCJACBJEH2OUSKZBUY6RVNMSXA3CM7Q, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.787618627 (378h58m34.658701672s) [noise=true , address=145.79.227.52:4445 , piece_id=YWZ6UHBVJWXLIGSE5HMV4WWZLLQLMBCB35WDSUFXK4XMBU5ZAT7Q, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.794650276 (378h58m34.665733311s) [noise=true , address=145.79.227.35:4482 , piece_id=3FUQ4OSF4B5DNWTAPY5W3LS2ZFBLCDWGXXLGASBKACSX3ONKLWWQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.794701083 (378h58m34.665784128s) [noise=true , address=145.79.227.39:4015 , piece_id=QB5S57M5HEYFEZOZYHBWMNFWVLQQEGHS73TBMANOJCCB42LK2AVA, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.833604161 (378h58m34.704687196s) [noise=true , address=145.79.227.48:4303 , piece_id=44PNGUZODJGAD7AFPZGJE6XBDYCSN2CLPIARGZI62PQ7KKU7ANWQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.857890877 (378h58m34.728973912s) [noise=true , address=145.79.227.27:4691 , piece_id=PEBFRJ6FC4PNFEODQJFGBHHC56RKLUECXWW36YBHNO47HDCDXJWQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.991414468 (378h58m34.862497503s) [noise=true , address=45.91.116.7:4011 , piece_id=UH4YHD7JFAKXOXJTNKRAUITFZVFWU5AJG5XHSLCJ6JS7JCLWEVRQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:35.048985192 (378h58m34.920068237s) [noise=true , address=45.91.116.14:4189 , piece_id=NF2SZ7V2LDV76ADGQ6NXVNBHTHNLMRYS4SXW6BT4OM7BQ5TQ6BPQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:35.055578377 (378h58m34.926661412s) [noise=true , address=45.91.116.8:4048 , piece_id=3G4VLJ76W4UGBDWZEIQCBDFPRGAM7QL4GMC23LS73BC3JSFGGHPA, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:35.055594557 (378h58m34.926677602s) [seg_pos={0 0}, upload=332595]: Segment reached optimal threshold of 13 pieces. Dec 21 10:27:35.055773338 (378h58m34.926856373s) [noise=true , address=45.91.116.4:4940 , piece_id=L6QW3XENG6UBHKUUXTMOQQRIXBWBN6XXL7SHLY7W5KC5ES3SFS3A, seg_pos={0 0}, upload=332595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 21 10:27:35.055779740 (378h58m34.926862775s) [noise=true , address=45.91.116.4:4940 , piece_id=L6QW3XENG6UBHKUUXTMOQQRIXBWBN6XXL7SHLY7W5KC5ES3SFS3A, seg_pos={0 0}, upload=332595]: Long tail cancellation due to optimal threshold Dec 21 10:27:35.055834805 (378h58m34.92691785s) [noise=true , address=77.90.31.15:4700 , piece_id=WQP7T5QHGW23C7I5TBY7PESM5YTTXS3BAEH3RDRQTJZYHI37BUJQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 21 10:27:35.055845194 (378h58m34.926928239s) [noise=true , address=77.90.31.15:4700 , piece_id=WQP7T5QHGW23C7I5TBY7PESM5YTTXS3BAEH3RDRQTJZYHI37BUJQ, seg_pos={0 0}, upload=332595]: Long tail cancellation due to optimal threshold Dec 21 10:27:35.055982826 (378h58m34.927065862s) [noise=true , address=45.91.116.12:4225 , piece_id=WH4P44CPP53IJXXMGK5VK2E3PYBGFLJBWSCFSCBKR2QTALIIEGEA, seg_pos={0 0}, upload=332595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 21 10:27:35.056041539 (378h58m34.927124594s) [noise=true , address=45.91.116.12:4225 , piece_id=WH4P44CPP53IJXXMGK5VK2E3PYBGFLJBWSCFSCBKR2QTALIIEGEA, seg_pos={0 0}, upload=332595]: Long tail cancellation due to optimal threshold Dec 21 10:27:35.056064683 (378h58m34.927147728s) [seg_pos={0 0}, upload=332595]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 10:27:35.056146969 (378h58m34.927230004s) [upload=332595]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 10:27:35.056153121 (378h58m34.927236176s) [upload=332595]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 10:27:35.353226207 (378h58m35.224309242s) [upload=332595]: Done uploading segments... Dec 21 10:27:35.353243942 (378h58m35.224326977s) [upload=332595]: Upload finished. err: <nil> Duration: 5.685065265s core uplink us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 740.834605ms trace: https://jaeger.dev.storj.io/trace/37011901b11802c9 FAIL (2025-12-21 10:52:20 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:37524->45.38.24.2:443: read: connection reset by peer Duration: 740.861937ms edge authservice http us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 24.282682757s trace: https://jaeger.dev.storj.io/trace/3ecb905698746e72 FAIL (2025-12-21 10:52:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: UH4YHD7JFAKXOXJTNKRAUITFZVFWU5AJG5XHSLCJ6JS7JCLWEVRQ) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 3G4VLJ76W4UGBDWZEIQCBDFPRGAM7QL4GMC23LS73BC3JSFGGHPA) context deadline exceeded ======LOGS====== Dec 21 10:27:32.520070050 (378h58m32.391153085s) [upload=332595]: Starting upload Dec 21 10:27:32.520134143 (378h58m32.391217178s) [upload=332595]: Done starting upload Dec 21 10:27:32.520194427 (378h58m32.391277472s) [upload=332595]: Uploading segments... Dec 21 10:27:32.520306451 (378h58m32.391389486s) [upload=332595]: Got next segment. Inline: false Dec 21 10:27:32.520335335 (378h58m32.39141839s) [upload=332595]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 10:27:32.520339694 (378h58m32.391422729s) [upload=332595]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 10:27:32.777396672 (378h58m32.648479718s) [seg_pos={0 0}, upload=332595]: Begin upload segment... Dec 21 10:27:32.777460795 (378h58m32.64854384s) [seg_pos={0 0}, upload=332595]: Done begin upload segment. Dec 21 10:27:32.777515899 (378h58m32.648598934s) [noise=true , address=145.79.227.23:4547 , piece_id=OOD2VMVRTCVGANLHQJME4HCJACBJEH2OUSKZBUY6RVNMSXA3CM7Q, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.777528564 (378h58m32.648611619s) [noise=true , address=145.79.227.27:4691 , piece_id=PEBFRJ6FC4PNFEODQJFGBHHC56RKLUECXWW36YBHNO47HDCDXJWQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.777605250 (378h58m32.648688295s) [noise=true , address=145.79.227.50:4373 , piece_id=GVHEJYGQ6A3HKPZ4V563E37K4L3CZUDEGCVJJHGNC722L3BNY46Q, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.778388482 (378h58m32.649471527s) [noise=true , address=145.79.227.39:4015 , piece_id=QB5S57M5HEYFEZOZYHBWMNFWVLQQEGHS73TBMANOJCCB42LK2AVA, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.778834370 (378h58m32.649917405s) [noise=true , address=45.91.116.12:4225 , piece_id=WH4P44CPP53IJXXMGK5VK2E3PYBGFLJBWSCFSCBKR2QTALIIEGEA, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.778858857 (378h58m32.649941892s) [noise=true , address=145.79.227.42:4121 , piece_id=FA7IZRDCG2XNN6PNDJOTZXGQXZWDOVV2XPR7MUSLK3YVYJVZE4UQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779235654 (378h58m32.650318689s) [noise=true , address=45.91.116.7:4011 , piece_id=UH4YHD7JFAKXOXJTNKRAUITFZVFWU5AJG5XHSLCJ6JS7JCLWEVRQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779269559 (378h58m32.650352594s) [noise=true , address=45.91.116.4:4940 , piece_id=L6QW3XENG6UBHKUUXTMOQQRIXBWBN6XXL7SHLY7W5KC5ES3SFS3A, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779536989 (378h58m32.650620024s) [noise=true , address=145.79.227.52:4445 , piece_id=YWZ6UHBVJWXLIGSE5HMV4WWZLLQLMBCB35WDSUFXK4XMBU5ZAT7Q, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779807954 (378h58m32.650890999s) [noise=true , address=77.90.31.15:4700 , piece_id=WQP7T5QHGW23C7I5TBY7PESM5YTTXS3BAEH3RDRQTJZYHI37BUJQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.780102345 (378h58m32.65118538s) [noise=true , address=145.79.227.17:4972 , piece_id=B3KYU5P3RL2BFR7QTS5ZK7B6UBDB3DTNTQKAGPM7B75S2SYEUFBQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.780482589 (378h58m32.651565634s) [noise=true , address=45.91.116.8:4048 , piece_id=3G4VLJ76W4UGBDWZEIQCBDFPRGAM7QL4GMC23LS73BC3JSFGGHPA, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.780863915 (378h58m32.65194696s) [noise=true , address=145.79.227.48:4303 , piece_id=44PNGUZODJGAD7AFPZGJE6XBDYCSN2CLPIARGZI62PQ7KKU7ANWQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.781783866 (378h58m32.652866911s) [noise=true , address=145.79.227.35:4482 , piece_id=3FUQ4OSF4B5DNWTAPY5W3LS2ZFBLCDWGXXLGASBKACSX3ONKLWWQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.782593317 (378h58m32.653676393s) [noise=true , address=45.91.116.14:4189 , piece_id=NF2SZ7V2LDV76ADGQ6NXVNBHTHNLMRYS4SXW6BT4OM7BQ5TQ6BPQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.783214190 (378h58m32.654297225s) [noise=true , address=145.79.227.51:4411 , piece_id=LC6PCUVNJN77OICI5JGXWN7LJD6JGMKKHHIQG6T4MKX5SU7FGRIQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:34.478418529 (378h58m34.349501564s) [upload=332595]: Next returned nil segment Dec 21 10:27:34.478430381 (378h58m34.349513416s) [upload=332595]: Waiting for error group managing segments... Dec 21 10:27:34.763620982 (378h58m34.634704027s) [noise=true , address=145.79.227.51:4411 , piece_id=LC6PCUVNJN77OICI5JGXWN7LJD6JGMKKHHIQG6T4MKX5SU7FGRIQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.770992478 (378h58m34.642075514s) [noise=true , address=145.79.227.50:4373 , piece_id=GVHEJYGQ6A3HKPZ4V563E37K4L3CZUDEGCVJJHGNC722L3BNY46Q, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.776530073 (378h58m34.647613108s) [noise=true , address=145.79.227.42:4121 , piece_id=FA7IZRDCG2XNN6PNDJOTZXGQXZWDOVV2XPR7MUSLK3YVYJVZE4UQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.786521177 (378h58m34.657604222s) [noise=true , address=145.79.227.17:4972 , piece_id=B3KYU5P3RL2BFR7QTS5ZK7B6UBDB3DTNTQKAGPM7B75S2SYEUFBQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.786591240 (378h58m34.657674275s) [noise=true , address=145.79.227.23:4547 , piece_id=OOD2VMVRTCVGANLHQJME4HCJACBJEH2OUSKZBUY6RVNMSXA3CM7Q, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.787618627 (378h58m34.658701672s) [noise=true , address=145.79.227.52:4445 , piece_id=YWZ6UHBVJWXLIGSE5HMV4WWZLLQLMBCB35WDSUFXK4XMBU5ZAT7Q, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.794650276 (378h58m34.665733311s) [noise=true , address=145.79.227.35:4482 , piece_id=3FUQ4OSF4B5DNWTAPY5W3LS2ZFBLCDWGXXLGASBKACSX3ONKLWWQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.794701083 (378h58m34.665784128s) [noise=true , address=145.79.227.39:4015 , piece_id=QB5S57M5HEYFEZOZYHBWMNFWVLQQEGHS73TBMANOJCCB42LK2AVA, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.833604161 (378h58m34.704687196s) [noise=true , address=145.79.227.48:4303 , piece_id=44PNGUZODJGAD7AFPZGJE6XBDYCSN2CLPIARGZI62PQ7KKU7ANWQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.857890877 (378h58m34.728973912s) [noise=true , address=145.79.227.27:4691 , piece_id=PEBFRJ6FC4PNFEODQJFGBHHC56RKLUECXWW36YBHNO47HDCDXJWQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.991414468 (378h58m34.862497503s) [noise=true , address=45.91.116.7:4011 , piece_id=UH4YHD7JFAKXOXJTNKRAUITFZVFWU5AJG5XHSLCJ6JS7JCLWEVRQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:35.048985192 (378h58m34.920068237s) [noise=true , address=45.91.116.14:4189 , piece_id=NF2SZ7V2LDV76ADGQ6NXVNBHTHNLMRYS4SXW6BT4OM7BQ5TQ6BPQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:35.055578377 (378h58m34.926661412s) [noise=true , address=45.91.116.8:4048 , piece_id=3G4VLJ76W4UGBDWZEIQCBDFPRGAM7QL4GMC23LS73BC3JSFGGHPA, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:35.055594557 (378h58m34.926677602s) [seg_pos={0 0}, upload=332595]: Segment reached optimal threshold of 13 pieces. Dec 21 10:27:35.055773338 (378h58m34.926856373s) [noise=true , address=45.91.116.4:4940 , piece_id=L6QW3XENG6UBHKUUXTMOQQRIXBWBN6XXL7SHLY7W5KC5ES3SFS3A, seg_pos={0 0}, upload=332595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 21 10:27:35.055779740 (378h58m34.926862775s) [noise=true , address=45.91.116.4:4940 , piece_id=L6QW3XENG6UBHKUUXTMOQQRIXBWBN6XXL7SHLY7W5KC5ES3SFS3A, seg_pos={0 0}, upload=332595]: Long tail cancellation due to optimal threshold Dec 21 10:27:35.055834805 (378h58m34.92691785s) [noise=true , address=77.90.31.15:4700 , piece_id=WQP7T5QHGW23C7I5TBY7PESM5YTTXS3BAEH3RDRQTJZYHI37BUJQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 21 10:27:35.055845194 (378h58m34.926928239s) [noise=true , address=77.90.31.15:4700 , piece_id=WQP7T5QHGW23C7I5TBY7PESM5YTTXS3BAEH3RDRQTJZYHI37BUJQ, seg_pos={0 0}, upload=332595]: Long tail cancellation due to optimal threshold Dec 21 10:27:35.055982826 (378h58m34.927065862s) [noise=true , address=45.91.116.12:4225 , piece_id=WH4P44CPP53IJXXMGK5VK2E3PYBGFLJBWSCFSCBKR2QTALIIEGEA, seg_pos={0 0}, upload=332595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 21 10:27:35.056041539 (378h58m34.927124594s) [noise=true , address=45.91.116.12:4225 , piece_id=WH4P44CPP53IJXXMGK5VK2E3PYBGFLJBWSCFSCBKR2QTALIIEGEA, seg_pos={0 0}, upload=332595]: Long tail cancellation due to optimal threshold Dec 21 10:27:35.056064683 (378h58m34.927147728s) [seg_pos={0 0}, upload=332595]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 10:27:35.056146969 (378h58m34.927230004s) [upload=332595]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 10:27:35.056153121 (378h58m34.927236176s) [upload=332595]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 10:27:35.353226207 (378h58m35.224309242s) [upload=332595]: Done uploading segments... Dec 21 10:27:35.353243942 (378h58m35.224326977s) [upload=332595]: Upload finished. err: <nil> Duration: 24.282706001s core uplink us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 726.147336ms trace: https://jaeger.dev.storj.io/trace/37011901b11802c9 FAIL (2025-12-21 10:51:46 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:37524->45.38.24.2:443: read: connection reset by peer Duration: 726.170971ms edge authservice http us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 6.020754929s trace: https://jaeger.dev.storj.io/trace/3ecb905698746e72 FAIL (2025-12-21 10:50:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: UH4YHD7JFAKXOXJTNKRAUITFZVFWU5AJG5XHSLCJ6JS7JCLWEVRQ) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 3G4VLJ76W4UGBDWZEIQCBDFPRGAM7QL4GMC23LS73BC3JSFGGHPA) context deadline exceeded ======LOGS====== Dec 21 10:27:32.520070050 (378h58m32.391153085s) [upload=332595]: Starting upload Dec 21 10:27:32.520134143 (378h58m32.391217178s) [upload=332595]: Done starting upload Dec 21 10:27:32.520194427 (378h58m32.391277472s) [upload=332595]: Uploading segments... Dec 21 10:27:32.520306451 (378h58m32.391389486s) [upload=332595]: Got next segment. Inline: false Dec 21 10:27:32.520335335 (378h58m32.39141839s) [upload=332595]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 10:27:32.520339694 (378h58m32.391422729s) [upload=332595]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 10:27:32.777396672 (378h58m32.648479718s) [seg_pos={0 0}, upload=332595]: Begin upload segment... Dec 21 10:27:32.777460795 (378h58m32.64854384s) [seg_pos={0 0}, upload=332595]: Done begin upload segment. Dec 21 10:27:32.777515899 (378h58m32.648598934s) [noise=true , address=145.79.227.23:4547 , piece_id=OOD2VMVRTCVGANLHQJME4HCJACBJEH2OUSKZBUY6RVNMSXA3CM7Q, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.777528564 (378h58m32.648611619s) [noise=true , address=145.79.227.27:4691 , piece_id=PEBFRJ6FC4PNFEODQJFGBHHC56RKLUECXWW36YBHNO47HDCDXJWQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.777605250 (378h58m32.648688295s) [noise=true , address=145.79.227.50:4373 , piece_id=GVHEJYGQ6A3HKPZ4V563E37K4L3CZUDEGCVJJHGNC722L3BNY46Q, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.778388482 (378h58m32.649471527s) [noise=true , address=145.79.227.39:4015 , piece_id=QB5S57M5HEYFEZOZYHBWMNFWVLQQEGHS73TBMANOJCCB42LK2AVA, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.778834370 (378h58m32.649917405s) [noise=true , address=45.91.116.12:4225 , piece_id=WH4P44CPP53IJXXMGK5VK2E3PYBGFLJBWSCFSCBKR2QTALIIEGEA, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.778858857 (378h58m32.649941892s) [noise=true , address=145.79.227.42:4121 , piece_id=FA7IZRDCG2XNN6PNDJOTZXGQXZWDOVV2XPR7MUSLK3YVYJVZE4UQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779235654 (378h58m32.650318689s) [noise=true , address=45.91.116.7:4011 , piece_id=UH4YHD7JFAKXOXJTNKRAUITFZVFWU5AJG5XHSLCJ6JS7JCLWEVRQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779269559 (378h58m32.650352594s) [noise=true , address=45.91.116.4:4940 , piece_id=L6QW3XENG6UBHKUUXTMOQQRIXBWBN6XXL7SHLY7W5KC5ES3SFS3A, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779536989 (378h58m32.650620024s) [noise=true , address=145.79.227.52:4445 , piece_id=YWZ6UHBVJWXLIGSE5HMV4WWZLLQLMBCB35WDSUFXK4XMBU5ZAT7Q, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779807954 (378h58m32.650890999s) [noise=true , address=77.90.31.15:4700 , piece_id=WQP7T5QHGW23C7I5TBY7PESM5YTTXS3BAEH3RDRQTJZYHI37BUJQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.780102345 (378h58m32.65118538s) [noise=true , address=145.79.227.17:4972 , piece_id=B3KYU5P3RL2BFR7QTS5ZK7B6UBDB3DTNTQKAGPM7B75S2SYEUFBQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.780482589 (378h58m32.651565634s) [noise=true , address=45.91.116.8:4048 , piece_id=3G4VLJ76W4UGBDWZEIQCBDFPRGAM7QL4GMC23LS73BC3JSFGGHPA, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.780863915 (378h58m32.65194696s) [noise=true , address=145.79.227.48:4303 , piece_id=44PNGUZODJGAD7AFPZGJE6XBDYCSN2CLPIARGZI62PQ7KKU7ANWQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.781783866 (378h58m32.652866911s) [noise=true , address=145.79.227.35:4482 , piece_id=3FUQ4OSF4B5DNWTAPY5W3LS2ZFBLCDWGXXLGASBKACSX3ONKLWWQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.782593317 (378h58m32.653676393s) [noise=true , address=45.91.116.14:4189 , piece_id=NF2SZ7V2LDV76ADGQ6NXVNBHTHNLMRYS4SXW6BT4OM7BQ5TQ6BPQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.783214190 (378h58m32.654297225s) [noise=true , address=145.79.227.51:4411 , piece_id=LC6PCUVNJN77OICI5JGXWN7LJD6JGMKKHHIQG6T4MKX5SU7FGRIQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:34.478418529 (378h58m34.349501564s) [upload=332595]: Next returned nil segment Dec 21 10:27:34.478430381 (378h58m34.349513416s) [upload=332595]: Waiting for error group managing segments... Dec 21 10:27:34.763620982 (378h58m34.634704027s) [noise=true , address=145.79.227.51:4411 , piece_id=LC6PCUVNJN77OICI5JGXWN7LJD6JGMKKHHIQG6T4MKX5SU7FGRIQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.770992478 (378h58m34.642075514s) [noise=true , address=145.79.227.50:4373 , piece_id=GVHEJYGQ6A3HKPZ4V563E37K4L3CZUDEGCVJJHGNC722L3BNY46Q, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.776530073 (378h58m34.647613108s) [noise=true , address=145.79.227.42:4121 , piece_id=FA7IZRDCG2XNN6PNDJOTZXGQXZWDOVV2XPR7MUSLK3YVYJVZE4UQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.786521177 (378h58m34.657604222s) [noise=true , address=145.79.227.17:4972 , piece_id=B3KYU5P3RL2BFR7QTS5ZK7B6UBDB3DTNTQKAGPM7B75S2SYEUFBQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.786591240 (378h58m34.657674275s) [noise=true , address=145.79.227.23:4547 , piece_id=OOD2VMVRTCVGANLHQJME4HCJACBJEH2OUSKZBUY6RVNMSXA3CM7Q, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.787618627 (378h58m34.658701672s) [noise=true , address=145.79.227.52:4445 , piece_id=YWZ6UHBVJWXLIGSE5HMV4WWZLLQLMBCB35WDSUFXK4XMBU5ZAT7Q, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.794650276 (378h58m34.665733311s) [noise=true , address=145.79.227.35:4482 , piece_id=3FUQ4OSF4B5DNWTAPY5W3LS2ZFBLCDWGXXLGASBKACSX3ONKLWWQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.794701083 (378h58m34.665784128s) [noise=true , address=145.79.227.39:4015 , piece_id=QB5S57M5HEYFEZOZYHBWMNFWVLQQEGHS73TBMANOJCCB42LK2AVA, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.833604161 (378h58m34.704687196s) [noise=true , address=145.79.227.48:4303 , piece_id=44PNGUZODJGAD7AFPZGJE6XBDYCSN2CLPIARGZI62PQ7KKU7ANWQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.857890877 (378h58m34.728973912s) [noise=true , address=145.79.227.27:4691 , piece_id=PEBFRJ6FC4PNFEODQJFGBHHC56RKLUECXWW36YBHNO47HDCDXJWQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.991414468 (378h58m34.862497503s) [noise=true , address=45.91.116.7:4011 , piece_id=UH4YHD7JFAKXOXJTNKRAUITFZVFWU5AJG5XHSLCJ6JS7JCLWEVRQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:35.048985192 (378h58m34.920068237s) [noise=true , address=45.91.116.14:4189 , piece_id=NF2SZ7V2LDV76ADGQ6NXVNBHTHNLMRYS4SXW6BT4OM7BQ5TQ6BPQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:35.055578377 (378h58m34.926661412s) [noise=true , address=45.91.116.8:4048 , piece_id=3G4VLJ76W4UGBDWZEIQCBDFPRGAM7QL4GMC23LS73BC3JSFGGHPA, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:35.055594557 (378h58m34.926677602s) [seg_pos={0 0}, upload=332595]: Segment reached optimal threshold of 13 pieces. Dec 21 10:27:35.055773338 (378h58m34.926856373s) [noise=true , address=45.91.116.4:4940 , piece_id=L6QW3XENG6UBHKUUXTMOQQRIXBWBN6XXL7SHLY7W5KC5ES3SFS3A, seg_pos={0 0}, upload=332595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 21 10:27:35.055779740 (378h58m34.926862775s) [noise=true , address=45.91.116.4:4940 , piece_id=L6QW3XENG6UBHKUUXTMOQQRIXBWBN6XXL7SHLY7W5KC5ES3SFS3A, seg_pos={0 0}, upload=332595]: Long tail cancellation due to optimal threshold Dec 21 10:27:35.055834805 (378h58m34.92691785s) [noise=true , address=77.90.31.15:4700 , piece_id=WQP7T5QHGW23C7I5TBY7PESM5YTTXS3BAEH3RDRQTJZYHI37BUJQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 21 10:27:35.055845194 (378h58m34.926928239s) [noise=true , address=77.90.31.15:4700 , piece_id=WQP7T5QHGW23C7I5TBY7PESM5YTTXS3BAEH3RDRQTJZYHI37BUJQ, seg_pos={0 0}, upload=332595]: Long tail cancellation due to optimal threshold Dec 21 10:27:35.055982826 (378h58m34.927065862s) [noise=true , address=45.91.116.12:4225 , piece_id=WH4P44CPP53IJXXMGK5VK2E3PYBGFLJBWSCFSCBKR2QTALIIEGEA, seg_pos={0 0}, upload=332595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 21 10:27:35.056041539 (378h58m34.927124594s) [noise=true , address=45.91.116.12:4225 , piece_id=WH4P44CPP53IJXXMGK5VK2E3PYBGFLJBWSCFSCBKR2QTALIIEGEA, seg_pos={0 0}, upload=332595]: Long tail cancellation due to optimal threshold Dec 21 10:27:35.056064683 (378h58m34.927147728s) [seg_pos={0 0}, upload=332595]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 10:27:35.056146969 (378h58m34.927230004s) [upload=332595]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 10:27:35.056153121 (378h58m34.927236176s) [upload=332595]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 10:27:35.353226207 (378h58m35.224309242s) [upload=332595]: Done uploading segments... Dec 21 10:27:35.353243942 (378h58m35.224326977s) [upload=332595]: Upload finished. err: <nil> Duration: 6.020781029s core uplink us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 727.038985ms trace: https://jaeger.dev.storj.io/trace/37011901b11802c9 FAIL (2025-12-21 10:50:52 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:37524->45.38.24.2:443: read: connection reset by peer Duration: 727.065666ms edge authservice http us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 713.470599ms trace: https://jaeger.dev.storj.io/trace/37011901b11802c9 FAIL (2025-12-21 10:49:57 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:37524->45.38.24.2:443: read: connection reset by peer Duration: 713.501257ms edge authservice http us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 5.786283535s trace: https://jaeger.dev.storj.io/trace/3ecb905698746e72 FAIL (2025-12-21 10:49:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: UH4YHD7JFAKXOXJTNKRAUITFZVFWU5AJG5XHSLCJ6JS7JCLWEVRQ) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 3G4VLJ76W4UGBDWZEIQCBDFPRGAM7QL4GMC23LS73BC3JSFGGHPA) context deadline exceeded ======LOGS====== Dec 21 10:27:32.520070050 (378h58m32.391153085s) [upload=332595]: Starting upload Dec 21 10:27:32.520134143 (378h58m32.391217178s) [upload=332595]: Done starting upload Dec 21 10:27:32.520194427 (378h58m32.391277472s) [upload=332595]: Uploading segments... Dec 21 10:27:32.520306451 (378h58m32.391389486s) [upload=332595]: Got next segment. Inline: false Dec 21 10:27:32.520335335 (378h58m32.39141839s) [upload=332595]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 10:27:32.520339694 (378h58m32.391422729s) [upload=332595]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 10:27:32.777396672 (378h58m32.648479718s) [seg_pos={0 0}, upload=332595]: Begin upload segment... Dec 21 10:27:32.777460795 (378h58m32.64854384s) [seg_pos={0 0}, upload=332595]: Done begin upload segment. Dec 21 10:27:32.777515899 (378h58m32.648598934s) [noise=true , address=145.79.227.23:4547 , piece_id=OOD2VMVRTCVGANLHQJME4HCJACBJEH2OUSKZBUY6RVNMSXA3CM7Q, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.777528564 (378h58m32.648611619s) [noise=true , address=145.79.227.27:4691 , piece_id=PEBFRJ6FC4PNFEODQJFGBHHC56RKLUECXWW36YBHNO47HDCDXJWQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.777605250 (378h58m32.648688295s) [noise=true , address=145.79.227.50:4373 , piece_id=GVHEJYGQ6A3HKPZ4V563E37K4L3CZUDEGCVJJHGNC722L3BNY46Q, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.778388482 (378h58m32.649471527s) [noise=true , address=145.79.227.39:4015 , piece_id=QB5S57M5HEYFEZOZYHBWMNFWVLQQEGHS73TBMANOJCCB42LK2AVA, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.778834370 (378h58m32.649917405s) [noise=true , address=45.91.116.12:4225 , piece_id=WH4P44CPP53IJXXMGK5VK2E3PYBGFLJBWSCFSCBKR2QTALIIEGEA, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.778858857 (378h58m32.649941892s) [noise=true , address=145.79.227.42:4121 , piece_id=FA7IZRDCG2XNN6PNDJOTZXGQXZWDOVV2XPR7MUSLK3YVYJVZE4UQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779235654 (378h58m32.650318689s) [noise=true , address=45.91.116.7:4011 , piece_id=UH4YHD7JFAKXOXJTNKRAUITFZVFWU5AJG5XHSLCJ6JS7JCLWEVRQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779269559 (378h58m32.650352594s) [noise=true , address=45.91.116.4:4940 , piece_id=L6QW3XENG6UBHKUUXTMOQQRIXBWBN6XXL7SHLY7W5KC5ES3SFS3A, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779536989 (378h58m32.650620024s) [noise=true , address=145.79.227.52:4445 , piece_id=YWZ6UHBVJWXLIGSE5HMV4WWZLLQLMBCB35WDSUFXK4XMBU5ZAT7Q, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779807954 (378h58m32.650890999s) [noise=true , address=77.90.31.15:4700 , piece_id=WQP7T5QHGW23C7I5TBY7PESM5YTTXS3BAEH3RDRQTJZYHI37BUJQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.780102345 (378h58m32.65118538s) [noise=true , address=145.79.227.17:4972 , piece_id=B3KYU5P3RL2BFR7QTS5ZK7B6UBDB3DTNTQKAGPM7B75S2SYEUFBQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.780482589 (378h58m32.651565634s) [noise=true , address=45.91.116.8:4048 , piece_id=3G4VLJ76W4UGBDWZEIQCBDFPRGAM7QL4GMC23LS73BC3JSFGGHPA, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.780863915 (378h58m32.65194696s) [noise=true , address=145.79.227.48:4303 , piece_id=44PNGUZODJGAD7AFPZGJE6XBDYCSN2CLPIARGZI62PQ7KKU7ANWQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.781783866 (378h58m32.652866911s) [noise=true , address=145.79.227.35:4482 , piece_id=3FUQ4OSF4B5DNWTAPY5W3LS2ZFBLCDWGXXLGASBKACSX3ONKLWWQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.782593317 (378h58m32.653676393s) [noise=true , address=45.91.116.14:4189 , piece_id=NF2SZ7V2LDV76ADGQ6NXVNBHTHNLMRYS4SXW6BT4OM7BQ5TQ6BPQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.783214190 (378h58m32.654297225s) [noise=true , address=145.79.227.51:4411 , piece_id=LC6PCUVNJN77OICI5JGXWN7LJD6JGMKKHHIQG6T4MKX5SU7FGRIQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:34.478418529 (378h58m34.349501564s) [upload=332595]: Next returned nil segment Dec 21 10:27:34.478430381 (378h58m34.349513416s) [upload=332595]: Waiting for error group managing segments... Dec 21 10:27:34.763620982 (378h58m34.634704027s) [noise=true , address=145.79.227.51:4411 , piece_id=LC6PCUVNJN77OICI5JGXWN7LJD6JGMKKHHIQG6T4MKX5SU7FGRIQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.770992478 (378h58m34.642075514s) [noise=true , address=145.79.227.50:4373 , piece_id=GVHEJYGQ6A3HKPZ4V563E37K4L3CZUDEGCVJJHGNC722L3BNY46Q, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.776530073 (378h58m34.647613108s) [noise=true , address=145.79.227.42:4121 , piece_id=FA7IZRDCG2XNN6PNDJOTZXGQXZWDOVV2XPR7MUSLK3YVYJVZE4UQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.786521177 (378h58m34.657604222s) [noise=true , address=145.79.227.17:4972 , piece_id=B3KYU5P3RL2BFR7QTS5ZK7B6UBDB3DTNTQKAGPM7B75S2SYEUFBQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.786591240 (378h58m34.657674275s) [noise=true , address=145.79.227.23:4547 , piece_id=OOD2VMVRTCVGANLHQJME4HCJACBJEH2OUSKZBUY6RVNMSXA3CM7Q, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.787618627 (378h58m34.658701672s) [noise=true , address=145.79.227.52:4445 , piece_id=YWZ6UHBVJWXLIGSE5HMV4WWZLLQLMBCB35WDSUFXK4XMBU5ZAT7Q, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.794650276 (378h58m34.665733311s) [noise=true , address=145.79.227.35:4482 , piece_id=3FUQ4OSF4B5DNWTAPY5W3LS2ZFBLCDWGXXLGASBKACSX3ONKLWWQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.794701083 (378h58m34.665784128s) [noise=true , address=145.79.227.39:4015 , piece_id=QB5S57M5HEYFEZOZYHBWMNFWVLQQEGHS73TBMANOJCCB42LK2AVA, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.833604161 (378h58m34.704687196s) [noise=true , address=145.79.227.48:4303 , piece_id=44PNGUZODJGAD7AFPZGJE6XBDYCSN2CLPIARGZI62PQ7KKU7ANWQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.857890877 (378h58m34.728973912s) [noise=true , address=145.79.227.27:4691 , piece_id=PEBFRJ6FC4PNFEODQJFGBHHC56RKLUECXWW36YBHNO47HDCDXJWQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.991414468 (378h58m34.862497503s) [noise=true , address=45.91.116.7:4011 , piece_id=UH4YHD7JFAKXOXJTNKRAUITFZVFWU5AJG5XHSLCJ6JS7JCLWEVRQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:35.048985192 (378h58m34.920068237s) [noise=true , address=45.91.116.14:4189 , piece_id=NF2SZ7V2LDV76ADGQ6NXVNBHTHNLMRYS4SXW6BT4OM7BQ5TQ6BPQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:35.055578377 (378h58m34.926661412s) [noise=true , address=45.91.116.8:4048 , piece_id=3G4VLJ76W4UGBDWZEIQCBDFPRGAM7QL4GMC23LS73BC3JSFGGHPA, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:35.055594557 (378h58m34.926677602s) [seg_pos={0 0}, upload=332595]: Segment reached optimal threshold of 13 pieces. Dec 21 10:27:35.055773338 (378h58m34.926856373s) [noise=true , address=45.91.116.4:4940 , piece_id=L6QW3XENG6UBHKUUXTMOQQRIXBWBN6XXL7SHLY7W5KC5ES3SFS3A, seg_pos={0 0}, upload=332595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 21 10:27:35.055779740 (378h58m34.926862775s) [noise=true , address=45.91.116.4:4940 , piece_id=L6QW3XENG6UBHKUUXTMOQQRIXBWBN6XXL7SHLY7W5KC5ES3SFS3A, seg_pos={0 0}, upload=332595]: Long tail cancellation due to optimal threshold Dec 21 10:27:35.055834805 (378h58m34.92691785s) [noise=true , address=77.90.31.15:4700 , piece_id=WQP7T5QHGW23C7I5TBY7PESM5YTTXS3BAEH3RDRQTJZYHI37BUJQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 21 10:27:35.055845194 (378h58m34.926928239s) [noise=true , address=77.90.31.15:4700 , piece_id=WQP7T5QHGW23C7I5TBY7PESM5YTTXS3BAEH3RDRQTJZYHI37BUJQ, seg_pos={0 0}, upload=332595]: Long tail cancellation due to optimal threshold Dec 21 10:27:35.055982826 (378h58m34.927065862s) [noise=true , address=45.91.116.12:4225 , piece_id=WH4P44CPP53IJXXMGK5VK2E3PYBGFLJBWSCFSCBKR2QTALIIEGEA, seg_pos={0 0}, upload=332595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 21 10:27:35.056041539 (378h58m34.927124594s) [noise=true , address=45.91.116.12:4225 , piece_id=WH4P44CPP53IJXXMGK5VK2E3PYBGFLJBWSCFSCBKR2QTALIIEGEA, seg_pos={0 0}, upload=332595]: Long tail cancellation due to optimal threshold Dec 21 10:27:35.056064683 (378h58m34.927147728s) [seg_pos={0 0}, upload=332595]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 10:27:35.056146969 (378h58m34.927230004s) [upload=332595]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 10:27:35.056153121 (378h58m34.927236176s) [upload=332595]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 10:27:35.353226207 (378h58m35.224309242s) [upload=332595]: Done uploading segments... Dec 21 10:27:35.353243942 (378h58m35.224326977s) [upload=332595]: Upload finished. err: <nil> Duration: 5.786300708s core uplink us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 728.024132ms trace: https://jaeger.dev.storj.io/trace/37011901b11802c9 FAIL (2025-12-21 10:49:16 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:37524->45.38.24.2:443: read: connection reset by peer Duration: 728.050442ms edge authservice http us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 691.469155ms trace: https://jaeger.dev.storj.io/trace/37011901b11802c9 FAIL (2025-12-21 10:48:58 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:37524->45.38.24.2:443: read: connection reset by peer Duration: 691.494524ms edge authservice http us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 5.686152616s trace: https://jaeger.dev.storj.io/trace/3ecb905698746e72 FAIL (2025-12-21 10:48:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: UH4YHD7JFAKXOXJTNKRAUITFZVFWU5AJG5XHSLCJ6JS7JCLWEVRQ) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 3G4VLJ76W4UGBDWZEIQCBDFPRGAM7QL4GMC23LS73BC3JSFGGHPA) context deadline exceeded ======LOGS====== Dec 21 10:27:32.520070050 (378h58m32.391153085s) [upload=332595]: Starting upload Dec 21 10:27:32.520134143 (378h58m32.391217178s) [upload=332595]: Done starting upload Dec 21 10:27:32.520194427 (378h58m32.391277472s) [upload=332595]: Uploading segments... Dec 21 10:27:32.520306451 (378h58m32.391389486s) [upload=332595]: Got next segment. Inline: false Dec 21 10:27:32.520335335 (378h58m32.39141839s) [upload=332595]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 10:27:32.520339694 (378h58m32.391422729s) [upload=332595]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 10:27:32.777396672 (378h58m32.648479718s) [seg_pos={0 0}, upload=332595]: Begin upload segment... Dec 21 10:27:32.777460795 (378h58m32.64854384s) [seg_pos={0 0}, upload=332595]: Done begin upload segment. Dec 21 10:27:32.777515899 (378h58m32.648598934s) [noise=true , address=145.79.227.23:4547 , piece_id=OOD2VMVRTCVGANLHQJME4HCJACBJEH2OUSKZBUY6RVNMSXA3CM7Q, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.777528564 (378h58m32.648611619s) [noise=true , address=145.79.227.27:4691 , piece_id=PEBFRJ6FC4PNFEODQJFGBHHC56RKLUECXWW36YBHNO47HDCDXJWQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.777605250 (378h58m32.648688295s) [noise=true , address=145.79.227.50:4373 , piece_id=GVHEJYGQ6A3HKPZ4V563E37K4L3CZUDEGCVJJHGNC722L3BNY46Q, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.778388482 (378h58m32.649471527s) [noise=true , address=145.79.227.39:4015 , piece_id=QB5S57M5HEYFEZOZYHBWMNFWVLQQEGHS73TBMANOJCCB42LK2AVA, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.778834370 (378h58m32.649917405s) [noise=true , address=45.91.116.12:4225 , piece_id=WH4P44CPP53IJXXMGK5VK2E3PYBGFLJBWSCFSCBKR2QTALIIEGEA, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.778858857 (378h58m32.649941892s) [noise=true , address=145.79.227.42:4121 , piece_id=FA7IZRDCG2XNN6PNDJOTZXGQXZWDOVV2XPR7MUSLK3YVYJVZE4UQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779235654 (378h58m32.650318689s) [noise=true , address=45.91.116.7:4011 , piece_id=UH4YHD7JFAKXOXJTNKRAUITFZVFWU5AJG5XHSLCJ6JS7JCLWEVRQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779269559 (378h58m32.650352594s) [noise=true , address=45.91.116.4:4940 , piece_id=L6QW3XENG6UBHKUUXTMOQQRIXBWBN6XXL7SHLY7W5KC5ES3SFS3A, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779536989 (378h58m32.650620024s) [noise=true , address=145.79.227.52:4445 , piece_id=YWZ6UHBVJWXLIGSE5HMV4WWZLLQLMBCB35WDSUFXK4XMBU5ZAT7Q, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779807954 (378h58m32.650890999s) [noise=true , address=77.90.31.15:4700 , piece_id=WQP7T5QHGW23C7I5TBY7PESM5YTTXS3BAEH3RDRQTJZYHI37BUJQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.780102345 (378h58m32.65118538s) [noise=true , address=145.79.227.17:4972 , piece_id=B3KYU5P3RL2BFR7QTS5ZK7B6UBDB3DTNTQKAGPM7B75S2SYEUFBQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.780482589 (378h58m32.651565634s) [noise=true , address=45.91.116.8:4048 , piece_id=3G4VLJ76W4UGBDWZEIQCBDFPRGAM7QL4GMC23LS73BC3JSFGGHPA, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.780863915 (378h58m32.65194696s) [noise=true , address=145.79.227.48:4303 , piece_id=44PNGUZODJGAD7AFPZGJE6XBDYCSN2CLPIARGZI62PQ7KKU7ANWQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.781783866 (378h58m32.652866911s) [noise=true , address=145.79.227.35:4482 , piece_id=3FUQ4OSF4B5DNWTAPY5W3LS2ZFBLCDWGXXLGASBKACSX3ONKLWWQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.782593317 (378h58m32.653676393s) [noise=true , address=45.91.116.14:4189 , piece_id=NF2SZ7V2LDV76ADGQ6NXVNBHTHNLMRYS4SXW6BT4OM7BQ5TQ6BPQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.783214190 (378h58m32.654297225s) [noise=true , address=145.79.227.51:4411 , piece_id=LC6PCUVNJN77OICI5JGXWN7LJD6JGMKKHHIQG6T4MKX5SU7FGRIQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:34.478418529 (378h58m34.349501564s) [upload=332595]: Next returned nil segment Dec 21 10:27:34.478430381 (378h58m34.349513416s) [upload=332595]: Waiting for error group managing segments... Dec 21 10:27:34.763620982 (378h58m34.634704027s) [noise=true , address=145.79.227.51:4411 , piece_id=LC6PCUVNJN77OICI5JGXWN7LJD6JGMKKHHIQG6T4MKX5SU7FGRIQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.770992478 (378h58m34.642075514s) [noise=true , address=145.79.227.50:4373 , piece_id=GVHEJYGQ6A3HKPZ4V563E37K4L3CZUDEGCVJJHGNC722L3BNY46Q, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.776530073 (378h58m34.647613108s) [noise=true , address=145.79.227.42:4121 , piece_id=FA7IZRDCG2XNN6PNDJOTZXGQXZWDOVV2XPR7MUSLK3YVYJVZE4UQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.786521177 (378h58m34.657604222s) [noise=true , address=145.79.227.17:4972 , piece_id=B3KYU5P3RL2BFR7QTS5ZK7B6UBDB3DTNTQKAGPM7B75S2SYEUFBQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.786591240 (378h58m34.657674275s) [noise=true , address=145.79.227.23:4547 , piece_id=OOD2VMVRTCVGANLHQJME4HCJACBJEH2OUSKZBUY6RVNMSXA3CM7Q, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.787618627 (378h58m34.658701672s) [noise=true , address=145.79.227.52:4445 , piece_id=YWZ6UHBVJWXLIGSE5HMV4WWZLLQLMBCB35WDSUFXK4XMBU5ZAT7Q, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.794650276 (378h58m34.665733311s) [noise=true , address=145.79.227.35:4482 , piece_id=3FUQ4OSF4B5DNWTAPY5W3LS2ZFBLCDWGXXLGASBKACSX3ONKLWWQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.794701083 (378h58m34.665784128s) [noise=true , address=145.79.227.39:4015 , piece_id=QB5S57M5HEYFEZOZYHBWMNFWVLQQEGHS73TBMANOJCCB42LK2AVA, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.833604161 (378h58m34.704687196s) [noise=true , address=145.79.227.48:4303 , piece_id=44PNGUZODJGAD7AFPZGJE6XBDYCSN2CLPIARGZI62PQ7KKU7ANWQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.857890877 (378h58m34.728973912s) [noise=true , address=145.79.227.27:4691 , piece_id=PEBFRJ6FC4PNFEODQJFGBHHC56RKLUECXWW36YBHNO47HDCDXJWQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.991414468 (378h58m34.862497503s) [noise=true , address=45.91.116.7:4011 , piece_id=UH4YHD7JFAKXOXJTNKRAUITFZVFWU5AJG5XHSLCJ6JS7JCLWEVRQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:35.048985192 (378h58m34.920068237s) [noise=true , address=45.91.116.14:4189 , piece_id=NF2SZ7V2LDV76ADGQ6NXVNBHTHNLMRYS4SXW6BT4OM7BQ5TQ6BPQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:35.055578377 (378h58m34.926661412s) [noise=true , address=45.91.116.8:4048 , piece_id=3G4VLJ76W4UGBDWZEIQCBDFPRGAM7QL4GMC23LS73BC3JSFGGHPA, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:35.055594557 (378h58m34.926677602s) [seg_pos={0 0}, upload=332595]: Segment reached optimal threshold of 13 pieces. Dec 21 10:27:35.055773338 (378h58m34.926856373s) [noise=true , address=45.91.116.4:4940 , piece_id=L6QW3XENG6UBHKUUXTMOQQRIXBWBN6XXL7SHLY7W5KC5ES3SFS3A, seg_pos={0 0}, upload=332595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 21 10:27:35.055779740 (378h58m34.926862775s) [noise=true , address=45.91.116.4:4940 , piece_id=L6QW3XENG6UBHKUUXTMOQQRIXBWBN6XXL7SHLY7W5KC5ES3SFS3A, seg_pos={0 0}, upload=332595]: Long tail cancellation due to optimal threshold Dec 21 10:27:35.055834805 (378h58m34.92691785s) [noise=true , address=77.90.31.15:4700 , piece_id=WQP7T5QHGW23C7I5TBY7PESM5YTTXS3BAEH3RDRQTJZYHI37BUJQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 21 10:27:35.055845194 (378h58m34.926928239s) [noise=true , address=77.90.31.15:4700 , piece_id=WQP7T5QHGW23C7I5TBY7PESM5YTTXS3BAEH3RDRQTJZYHI37BUJQ, seg_pos={0 0}, upload=332595]: Long tail cancellation due to optimal threshold Dec 21 10:27:35.055982826 (378h58m34.927065862s) [noise=true , address=45.91.116.12:4225 , piece_id=WH4P44CPP53IJXXMGK5VK2E3PYBGFLJBWSCFSCBKR2QTALIIEGEA, seg_pos={0 0}, upload=332595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 21 10:27:35.056041539 (378h58m34.927124594s) [noise=true , address=45.91.116.12:4225 , piece_id=WH4P44CPP53IJXXMGK5VK2E3PYBGFLJBWSCFSCBKR2QTALIIEGEA, seg_pos={0 0}, upload=332595]: Long tail cancellation due to optimal threshold Dec 21 10:27:35.056064683 (378h58m34.927147728s) [seg_pos={0 0}, upload=332595]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 10:27:35.056146969 (378h58m34.927230004s) [upload=332595]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 10:27:35.056153121 (378h58m34.927236176s) [upload=332595]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 10:27:35.353226207 (378h58m35.224309242s) [upload=332595]: Done uploading segments... Dec 21 10:27:35.353243942 (378h58m35.224326977s) [upload=332595]: Upload finished. err: <nil> Duration: 5.686182274s core uplink us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 754.459357ms trace: https://jaeger.dev.storj.io/trace/37011901b11802c9 FAIL (2025-12-21 10:48:30 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:37524->45.38.24.2:443: read: connection reset by peer Duration: 754.479095ms edge authservice http us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 6.462673038s trace: https://jaeger.dev.storj.io/trace/3ecb905698746e72 FAIL (2025-12-21 10:48:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: UH4YHD7JFAKXOXJTNKRAUITFZVFWU5AJG5XHSLCJ6JS7JCLWEVRQ) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 3G4VLJ76W4UGBDWZEIQCBDFPRGAM7QL4GMC23LS73BC3JSFGGHPA) context deadline exceeded ======LOGS====== Dec 21 10:27:32.520070050 (378h58m32.391153085s) [upload=332595]: Starting upload Dec 21 10:27:32.520134143 (378h58m32.391217178s) [upload=332595]: Done starting upload Dec 21 10:27:32.520194427 (378h58m32.391277472s) [upload=332595]: Uploading segments... Dec 21 10:27:32.520306451 (378h58m32.391389486s) [upload=332595]: Got next segment. Inline: false Dec 21 10:27:32.520335335 (378h58m32.39141839s) [upload=332595]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 10:27:32.520339694 (378h58m32.391422729s) [upload=332595]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 10:27:32.777396672 (378h58m32.648479718s) [seg_pos={0 0}, upload=332595]: Begin upload segment... Dec 21 10:27:32.777460795 (378h58m32.64854384s) [seg_pos={0 0}, upload=332595]: Done begin upload segment. Dec 21 10:27:32.777515899 (378h58m32.648598934s) [noise=true , address=145.79.227.23:4547 , piece_id=OOD2VMVRTCVGANLHQJME4HCJACBJEH2OUSKZBUY6RVNMSXA3CM7Q, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.777528564 (378h58m32.648611619s) [noise=true , address=145.79.227.27:4691 , piece_id=PEBFRJ6FC4PNFEODQJFGBHHC56RKLUECXWW36YBHNO47HDCDXJWQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.777605250 (378h58m32.648688295s) [noise=true , address=145.79.227.50:4373 , piece_id=GVHEJYGQ6A3HKPZ4V563E37K4L3CZUDEGCVJJHGNC722L3BNY46Q, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.778388482 (378h58m32.649471527s) [noise=true , address=145.79.227.39:4015 , piece_id=QB5S57M5HEYFEZOZYHBWMNFWVLQQEGHS73TBMANOJCCB42LK2AVA, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.778834370 (378h58m32.649917405s) [noise=true , address=45.91.116.12:4225 , piece_id=WH4P44CPP53IJXXMGK5VK2E3PYBGFLJBWSCFSCBKR2QTALIIEGEA, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.778858857 (378h58m32.649941892s) [noise=true , address=145.79.227.42:4121 , piece_id=FA7IZRDCG2XNN6PNDJOTZXGQXZWDOVV2XPR7MUSLK3YVYJVZE4UQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779235654 (378h58m32.650318689s) [noise=true , address=45.91.116.7:4011 , piece_id=UH4YHD7JFAKXOXJTNKRAUITFZVFWU5AJG5XHSLCJ6JS7JCLWEVRQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779269559 (378h58m32.650352594s) [noise=true , address=45.91.116.4:4940 , piece_id=L6QW3XENG6UBHKUUXTMOQQRIXBWBN6XXL7SHLY7W5KC5ES3SFS3A, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779536989 (378h58m32.650620024s) [noise=true , address=145.79.227.52:4445 , piece_id=YWZ6UHBVJWXLIGSE5HMV4WWZLLQLMBCB35WDSUFXK4XMBU5ZAT7Q, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779807954 (378h58m32.650890999s) [noise=true , address=77.90.31.15:4700 , piece_id=WQP7T5QHGW23C7I5TBY7PESM5YTTXS3BAEH3RDRQTJZYHI37BUJQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.780102345 (378h58m32.65118538s) [noise=true , address=145.79.227.17:4972 , piece_id=B3KYU5P3RL2BFR7QTS5ZK7B6UBDB3DTNTQKAGPM7B75S2SYEUFBQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.780482589 (378h58m32.651565634s) [noise=true , address=45.91.116.8:4048 , piece_id=3G4VLJ76W4UGBDWZEIQCBDFPRGAM7QL4GMC23LS73BC3JSFGGHPA, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.780863915 (378h58m32.65194696s) [noise=true , address=145.79.227.48:4303 , piece_id=44PNGUZODJGAD7AFPZGJE6XBDYCSN2CLPIARGZI62PQ7KKU7ANWQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.781783866 (378h58m32.652866911s) [noise=true , address=145.79.227.35:4482 , piece_id=3FUQ4OSF4B5DNWTAPY5W3LS2ZFBLCDWGXXLGASBKACSX3ONKLWWQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.782593317 (378h58m32.653676393s) [noise=true , address=45.91.116.14:4189 , piece_id=NF2SZ7V2LDV76ADGQ6NXVNBHTHNLMRYS4SXW6BT4OM7BQ5TQ6BPQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.783214190 (378h58m32.654297225s) [noise=true , address=145.79.227.51:4411 , piece_id=LC6PCUVNJN77OICI5JGXWN7LJD6JGMKKHHIQG6T4MKX5SU7FGRIQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:34.478418529 (378h58m34.349501564s) [upload=332595]: Next returned nil segment Dec 21 10:27:34.478430381 (378h58m34.349513416s) [upload=332595]: Waiting for error group managing segments... Dec 21 10:27:34.763620982 (378h58m34.634704027s) [noise=true , address=145.79.227.51:4411 , piece_id=LC6PCUVNJN77OICI5JGXWN7LJD6JGMKKHHIQG6T4MKX5SU7FGRIQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.770992478 (378h58m34.642075514s) [noise=true , address=145.79.227.50:4373 , piece_id=GVHEJYGQ6A3HKPZ4V563E37K4L3CZUDEGCVJJHGNC722L3BNY46Q, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.776530073 (378h58m34.647613108s) [noise=true , address=145.79.227.42:4121 , piece_id=FA7IZRDCG2XNN6PNDJOTZXGQXZWDOVV2XPR7MUSLK3YVYJVZE4UQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.786521177 (378h58m34.657604222s) [noise=true , address=145.79.227.17:4972 , piece_id=B3KYU5P3RL2BFR7QTS5ZK7B6UBDB3DTNTQKAGPM7B75S2SYEUFBQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.786591240 (378h58m34.657674275s) [noise=true , address=145.79.227.23:4547 , piece_id=OOD2VMVRTCVGANLHQJME4HCJACBJEH2OUSKZBUY6RVNMSXA3CM7Q, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.787618627 (378h58m34.658701672s) [noise=true , address=145.79.227.52:4445 , piece_id=YWZ6UHBVJWXLIGSE5HMV4WWZLLQLMBCB35WDSUFXK4XMBU5ZAT7Q, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.794650276 (378h58m34.665733311s) [noise=true , address=145.79.227.35:4482 , piece_id=3FUQ4OSF4B5DNWTAPY5W3LS2ZFBLCDWGXXLGASBKACSX3ONKLWWQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.794701083 (378h58m34.665784128s) [noise=true , address=145.79.227.39:4015 , piece_id=QB5S57M5HEYFEZOZYHBWMNFWVLQQEGHS73TBMANOJCCB42LK2AVA, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.833604161 (378h58m34.704687196s) [noise=true , address=145.79.227.48:4303 , piece_id=44PNGUZODJGAD7AFPZGJE6XBDYCSN2CLPIARGZI62PQ7KKU7ANWQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.857890877 (378h58m34.728973912s) [noise=true , address=145.79.227.27:4691 , piece_id=PEBFRJ6FC4PNFEODQJFGBHHC56RKLUECXWW36YBHNO47HDCDXJWQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.991414468 (378h58m34.862497503s) [noise=true , address=45.91.116.7:4011 , piece_id=UH4YHD7JFAKXOXJTNKRAUITFZVFWU5AJG5XHSLCJ6JS7JCLWEVRQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:35.048985192 (378h58m34.920068237s) [noise=true , address=45.91.116.14:4189 , piece_id=NF2SZ7V2LDV76ADGQ6NXVNBHTHNLMRYS4SXW6BT4OM7BQ5TQ6BPQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:35.055578377 (378h58m34.926661412s) [noise=true , address=45.91.116.8:4048 , piece_id=3G4VLJ76W4UGBDWZEIQCBDFPRGAM7QL4GMC23LS73BC3JSFGGHPA, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:35.055594557 (378h58m34.926677602s) [seg_pos={0 0}, upload=332595]: Segment reached optimal threshold of 13 pieces. Dec 21 10:27:35.055773338 (378h58m34.926856373s) [noise=true , address=45.91.116.4:4940 , piece_id=L6QW3XENG6UBHKUUXTMOQQRIXBWBN6XXL7SHLY7W5KC5ES3SFS3A, seg_pos={0 0}, upload=332595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 21 10:27:35.055779740 (378h58m34.926862775s) [noise=true , address=45.91.116.4:4940 , piece_id=L6QW3XENG6UBHKUUXTMOQQRIXBWBN6XXL7SHLY7W5KC5ES3SFS3A, seg_pos={0 0}, upload=332595]: Long tail cancellation due to optimal threshold Dec 21 10:27:35.055834805 (378h58m34.92691785s) [noise=true , address=77.90.31.15:4700 , piece_id=WQP7T5QHGW23C7I5TBY7PESM5YTTXS3BAEH3RDRQTJZYHI37BUJQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 21 10:27:35.055845194 (378h58m34.926928239s) [noise=true , address=77.90.31.15:4700 , piece_id=WQP7T5QHGW23C7I5TBY7PESM5YTTXS3BAEH3RDRQTJZYHI37BUJQ, seg_pos={0 0}, upload=332595]: Long tail cancellation due to optimal threshold Dec 21 10:27:35.055982826 (378h58m34.927065862s) [noise=true , address=45.91.116.12:4225 , piece_id=WH4P44CPP53IJXXMGK5VK2E3PYBGFLJBWSCFSCBKR2QTALIIEGEA, seg_pos={0 0}, upload=332595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 21 10:27:35.056041539 (378h58m34.927124594s) [noise=true , address=45.91.116.12:4225 , piece_id=WH4P44CPP53IJXXMGK5VK2E3PYBGFLJBWSCFSCBKR2QTALIIEGEA, seg_pos={0 0}, upload=332595]: Long tail cancellation due to optimal threshold Dec 21 10:27:35.056064683 (378h58m34.927147728s) [seg_pos={0 0}, upload=332595]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 10:27:35.056146969 (378h58m34.927230004s) [upload=332595]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 10:27:35.056153121 (378h58m34.927236176s) [upload=332595]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 10:27:35.353226207 (378h58m35.224309242s) [upload=332595]: Done uploading segments... Dec 21 10:27:35.353243942 (378h58m35.224326977s) [upload=332595]: Upload finished. err: <nil> Duration: 6.462700932s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 5.738078982s trace: https://jaeger.dev.storj.io/trace/3ecb905698746e72 FAIL (2025-12-21 10:47:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: UH4YHD7JFAKXOXJTNKRAUITFZVFWU5AJG5XHSLCJ6JS7JCLWEVRQ) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 3G4VLJ76W4UGBDWZEIQCBDFPRGAM7QL4GMC23LS73BC3JSFGGHPA) context deadline exceeded ======LOGS====== Dec 21 10:27:32.520070050 (378h58m32.391153085s) [upload=332595]: Starting upload Dec 21 10:27:32.520134143 (378h58m32.391217178s) [upload=332595]: Done starting upload Dec 21 10:27:32.520194427 (378h58m32.391277472s) [upload=332595]: Uploading segments... Dec 21 10:27:32.520306451 (378h58m32.391389486s) [upload=332595]: Got next segment. Inline: false Dec 21 10:27:32.520335335 (378h58m32.39141839s) [upload=332595]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 10:27:32.520339694 (378h58m32.391422729s) [upload=332595]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 10:27:32.777396672 (378h58m32.648479718s) [seg_pos={0 0}, upload=332595]: Begin upload segment... Dec 21 10:27:32.777460795 (378h58m32.64854384s) [seg_pos={0 0}, upload=332595]: Done begin upload segment. Dec 21 10:27:32.777515899 (378h58m32.648598934s) [noise=true , address=145.79.227.23:4547 , piece_id=OOD2VMVRTCVGANLHQJME4HCJACBJEH2OUSKZBUY6RVNMSXA3CM7Q, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.777528564 (378h58m32.648611619s) [noise=true , address=145.79.227.27:4691 , piece_id=PEBFRJ6FC4PNFEODQJFGBHHC56RKLUECXWW36YBHNO47HDCDXJWQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.777605250 (378h58m32.648688295s) [noise=true , address=145.79.227.50:4373 , piece_id=GVHEJYGQ6A3HKPZ4V563E37K4L3CZUDEGCVJJHGNC722L3BNY46Q, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.778388482 (378h58m32.649471527s) [noise=true , address=145.79.227.39:4015 , piece_id=QB5S57M5HEYFEZOZYHBWMNFWVLQQEGHS73TBMANOJCCB42LK2AVA, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.778834370 (378h58m32.649917405s) [noise=true , address=45.91.116.12:4225 , piece_id=WH4P44CPP53IJXXMGK5VK2E3PYBGFLJBWSCFSCBKR2QTALIIEGEA, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.778858857 (378h58m32.649941892s) [noise=true , address=145.79.227.42:4121 , piece_id=FA7IZRDCG2XNN6PNDJOTZXGQXZWDOVV2XPR7MUSLK3YVYJVZE4UQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779235654 (378h58m32.650318689s) [noise=true , address=45.91.116.7:4011 , piece_id=UH4YHD7JFAKXOXJTNKRAUITFZVFWU5AJG5XHSLCJ6JS7JCLWEVRQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779269559 (378h58m32.650352594s) [noise=true , address=45.91.116.4:4940 , piece_id=L6QW3XENG6UBHKUUXTMOQQRIXBWBN6XXL7SHLY7W5KC5ES3SFS3A, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779536989 (378h58m32.650620024s) [noise=true , address=145.79.227.52:4445 , piece_id=YWZ6UHBVJWXLIGSE5HMV4WWZLLQLMBCB35WDSUFXK4XMBU5ZAT7Q, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779807954 (378h58m32.650890999s) [noise=true , address=77.90.31.15:4700 , piece_id=WQP7T5QHGW23C7I5TBY7PESM5YTTXS3BAEH3RDRQTJZYHI37BUJQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.780102345 (378h58m32.65118538s) [noise=true , address=145.79.227.17:4972 , piece_id=B3KYU5P3RL2BFR7QTS5ZK7B6UBDB3DTNTQKAGPM7B75S2SYEUFBQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.780482589 (378h58m32.651565634s) [noise=true , address=45.91.116.8:4048 , piece_id=3G4VLJ76W4UGBDWZEIQCBDFPRGAM7QL4GMC23LS73BC3JSFGGHPA, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.780863915 (378h58m32.65194696s) [noise=true , address=145.79.227.48:4303 , piece_id=44PNGUZODJGAD7AFPZGJE6XBDYCSN2CLPIARGZI62PQ7KKU7ANWQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.781783866 (378h58m32.652866911s) [noise=true , address=145.79.227.35:4482 , piece_id=3FUQ4OSF4B5DNWTAPY5W3LS2ZFBLCDWGXXLGASBKACSX3ONKLWWQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.782593317 (378h58m32.653676393s) [noise=true , address=45.91.116.14:4189 , piece_id=NF2SZ7V2LDV76ADGQ6NXVNBHTHNLMRYS4SXW6BT4OM7BQ5TQ6BPQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.783214190 (378h58m32.654297225s) [noise=true , address=145.79.227.51:4411 , piece_id=LC6PCUVNJN77OICI5JGXWN7LJD6JGMKKHHIQG6T4MKX5SU7FGRIQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:34.478418529 (378h58m34.349501564s) [upload=332595]: Next returned nil segment Dec 21 10:27:34.478430381 (378h58m34.349513416s) [upload=332595]: Waiting for error group managing segments... Dec 21 10:27:34.763620982 (378h58m34.634704027s) [noise=true , address=145.79.227.51:4411 , piece_id=LC6PCUVNJN77OICI5JGXWN7LJD6JGMKKHHIQG6T4MKX5SU7FGRIQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.770992478 (378h58m34.642075514s) [noise=true , address=145.79.227.50:4373 , piece_id=GVHEJYGQ6A3HKPZ4V563E37K4L3CZUDEGCVJJHGNC722L3BNY46Q, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.776530073 (378h58m34.647613108s) [noise=true , address=145.79.227.42:4121 , piece_id=FA7IZRDCG2XNN6PNDJOTZXGQXZWDOVV2XPR7MUSLK3YVYJVZE4UQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.786521177 (378h58m34.657604222s) [noise=true , address=145.79.227.17:4972 , piece_id=B3KYU5P3RL2BFR7QTS5ZK7B6UBDB3DTNTQKAGPM7B75S2SYEUFBQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.786591240 (378h58m34.657674275s) [noise=true , address=145.79.227.23:4547 , piece_id=OOD2VMVRTCVGANLHQJME4HCJACBJEH2OUSKZBUY6RVNMSXA3CM7Q, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.787618627 (378h58m34.658701672s) [noise=true , address=145.79.227.52:4445 , piece_id=YWZ6UHBVJWXLIGSE5HMV4WWZLLQLMBCB35WDSUFXK4XMBU5ZAT7Q, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.794650276 (378h58m34.665733311s) [noise=true , address=145.79.227.35:4482 , piece_id=3FUQ4OSF4B5DNWTAPY5W3LS2ZFBLCDWGXXLGASBKACSX3ONKLWWQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.794701083 (378h58m34.665784128s) [noise=true , address=145.79.227.39:4015 , piece_id=QB5S57M5HEYFEZOZYHBWMNFWVLQQEGHS73TBMANOJCCB42LK2AVA, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.833604161 (378h58m34.704687196s) [noise=true , address=145.79.227.48:4303 , piece_id=44PNGUZODJGAD7AFPZGJE6XBDYCSN2CLPIARGZI62PQ7KKU7ANWQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.857890877 (378h58m34.728973912s) [noise=true , address=145.79.227.27:4691 , piece_id=PEBFRJ6FC4PNFEODQJFGBHHC56RKLUECXWW36YBHNO47HDCDXJWQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.991414468 (378h58m34.862497503s) [noise=true , address=45.91.116.7:4011 , piece_id=UH4YHD7JFAKXOXJTNKRAUITFZVFWU5AJG5XHSLCJ6JS7JCLWEVRQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:35.048985192 (378h58m34.920068237s) [noise=true , address=45.91.116.14:4189 , piece_id=NF2SZ7V2LDV76ADGQ6NXVNBHTHNLMRYS4SXW6BT4OM7BQ5TQ6BPQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:35.055578377 (378h58m34.926661412s) [noise=true , address=45.91.116.8:4048 , piece_id=3G4VLJ76W4UGBDWZEIQCBDFPRGAM7QL4GMC23LS73BC3JSFGGHPA, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:35.055594557 (378h58m34.926677602s) [seg_pos={0 0}, upload=332595]: Segment reached optimal threshold of 13 pieces. Dec 21 10:27:35.055773338 (378h58m34.926856373s) [noise=true , address=45.91.116.4:4940 , piece_id=L6QW3XENG6UBHKUUXTMOQQRIXBWBN6XXL7SHLY7W5KC5ES3SFS3A, seg_pos={0 0}, upload=332595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 21 10:27:35.055779740 (378h58m34.926862775s) [noise=true , address=45.91.116.4:4940 , piece_id=L6QW3XENG6UBHKUUXTMOQQRIXBWBN6XXL7SHLY7W5KC5ES3SFS3A, seg_pos={0 0}, upload=332595]: Long tail cancellation due to optimal threshold Dec 21 10:27:35.055834805 (378h58m34.92691785s) [noise=true , address=77.90.31.15:4700 , piece_id=WQP7T5QHGW23C7I5TBY7PESM5YTTXS3BAEH3RDRQTJZYHI37BUJQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 21 10:27:35.055845194 (378h58m34.926928239s) [noise=true , address=77.90.31.15:4700 , piece_id=WQP7T5QHGW23C7I5TBY7PESM5YTTXS3BAEH3RDRQTJZYHI37BUJQ, seg_pos={0 0}, upload=332595]: Long tail cancellation due to optimal threshold Dec 21 10:27:35.055982826 (378h58m34.927065862s) [noise=true , address=45.91.116.12:4225 , piece_id=WH4P44CPP53IJXXMGK5VK2E3PYBGFLJBWSCFSCBKR2QTALIIEGEA, seg_pos={0 0}, upload=332595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 21 10:27:35.056041539 (378h58m34.927124594s) [noise=true , address=45.91.116.12:4225 , piece_id=WH4P44CPP53IJXXMGK5VK2E3PYBGFLJBWSCFSCBKR2QTALIIEGEA, seg_pos={0 0}, upload=332595]: Long tail cancellation due to optimal threshold Dec 21 10:27:35.056064683 (378h58m34.927147728s) [seg_pos={0 0}, upload=332595]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 10:27:35.056146969 (378h58m34.927230004s) [upload=332595]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 10:27:35.056153121 (378h58m34.927236176s) [upload=332595]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 10:27:35.353226207 (378h58m35.224309242s) [upload=332595]: Done uploading segments... Dec 21 10:27:35.353243942 (378h58m35.224326977s) [upload=332595]: Upload finished. err: <nil> Duration: 5.738112365s core uplink us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 841.397712ms trace: https://jaeger.dev.storj.io/trace/37011901b11802c9 FAIL (2025-12-21 10:47:42 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:37524->45.38.24.2:443: read: connection reset by peer Duration: 841.420155ms edge authservice http us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 5.699846609s trace: https://jaeger.dev.storj.io/trace/3ecb905698746e72 FAIL (2025-12-21 10:46:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: UH4YHD7JFAKXOXJTNKRAUITFZVFWU5AJG5XHSLCJ6JS7JCLWEVRQ) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 3G4VLJ76W4UGBDWZEIQCBDFPRGAM7QL4GMC23LS73BC3JSFGGHPA) context deadline exceeded ======LOGS====== Dec 21 10:27:32.520070050 (378h58m32.391153085s) [upload=332595]: Starting upload Dec 21 10:27:32.520134143 (378h58m32.391217178s) [upload=332595]: Done starting upload Dec 21 10:27:32.520194427 (378h58m32.391277472s) [upload=332595]: Uploading segments... Dec 21 10:27:32.520306451 (378h58m32.391389486s) [upload=332595]: Got next segment. Inline: false Dec 21 10:27:32.520335335 (378h58m32.39141839s) [upload=332595]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 10:27:32.520339694 (378h58m32.391422729s) [upload=332595]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 10:27:32.777396672 (378h58m32.648479718s) [seg_pos={0 0}, upload=332595]: Begin upload segment... Dec 21 10:27:32.777460795 (378h58m32.64854384s) [seg_pos={0 0}, upload=332595]: Done begin upload segment. Dec 21 10:27:32.777515899 (378h58m32.648598934s) [noise=true , address=145.79.227.23:4547 , piece_id=OOD2VMVRTCVGANLHQJME4HCJACBJEH2OUSKZBUY6RVNMSXA3CM7Q, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.777528564 (378h58m32.648611619s) [noise=true , address=145.79.227.27:4691 , piece_id=PEBFRJ6FC4PNFEODQJFGBHHC56RKLUECXWW36YBHNO47HDCDXJWQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.777605250 (378h58m32.648688295s) [noise=true , address=145.79.227.50:4373 , piece_id=GVHEJYGQ6A3HKPZ4V563E37K4L3CZUDEGCVJJHGNC722L3BNY46Q, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.778388482 (378h58m32.649471527s) [noise=true , address=145.79.227.39:4015 , piece_id=QB5S57M5HEYFEZOZYHBWMNFWVLQQEGHS73TBMANOJCCB42LK2AVA, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.778834370 (378h58m32.649917405s) [noise=true , address=45.91.116.12:4225 , piece_id=WH4P44CPP53IJXXMGK5VK2E3PYBGFLJBWSCFSCBKR2QTALIIEGEA, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.778858857 (378h58m32.649941892s) [noise=true , address=145.79.227.42:4121 , piece_id=FA7IZRDCG2XNN6PNDJOTZXGQXZWDOVV2XPR7MUSLK3YVYJVZE4UQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779235654 (378h58m32.650318689s) [noise=true , address=45.91.116.7:4011 , piece_id=UH4YHD7JFAKXOXJTNKRAUITFZVFWU5AJG5XHSLCJ6JS7JCLWEVRQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779269559 (378h58m32.650352594s) [noise=true , address=45.91.116.4:4940 , piece_id=L6QW3XENG6UBHKUUXTMOQQRIXBWBN6XXL7SHLY7W5KC5ES3SFS3A, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779536989 (378h58m32.650620024s) [noise=true , address=145.79.227.52:4445 , piece_id=YWZ6UHBVJWXLIGSE5HMV4WWZLLQLMBCB35WDSUFXK4XMBU5ZAT7Q, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779807954 (378h58m32.650890999s) [noise=true , address=77.90.31.15:4700 , piece_id=WQP7T5QHGW23C7I5TBY7PESM5YTTXS3BAEH3RDRQTJZYHI37BUJQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.780102345 (378h58m32.65118538s) [noise=true , address=145.79.227.17:4972 , piece_id=B3KYU5P3RL2BFR7QTS5ZK7B6UBDB3DTNTQKAGPM7B75S2SYEUFBQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.780482589 (378h58m32.651565634s) [noise=true , address=45.91.116.8:4048 , piece_id=3G4VLJ76W4UGBDWZEIQCBDFPRGAM7QL4GMC23LS73BC3JSFGGHPA, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.780863915 (378h58m32.65194696s) [noise=true , address=145.79.227.48:4303 , piece_id=44PNGUZODJGAD7AFPZGJE6XBDYCSN2CLPIARGZI62PQ7KKU7ANWQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.781783866 (378h58m32.652866911s) [noise=true , address=145.79.227.35:4482 , piece_id=3FUQ4OSF4B5DNWTAPY5W3LS2ZFBLCDWGXXLGASBKACSX3ONKLWWQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.782593317 (378h58m32.653676393s) [noise=true , address=45.91.116.14:4189 , piece_id=NF2SZ7V2LDV76ADGQ6NXVNBHTHNLMRYS4SXW6BT4OM7BQ5TQ6BPQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.783214190 (378h58m32.654297225s) [noise=true , address=145.79.227.51:4411 , piece_id=LC6PCUVNJN77OICI5JGXWN7LJD6JGMKKHHIQG6T4MKX5SU7FGRIQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:34.478418529 (378h58m34.349501564s) [upload=332595]: Next returned nil segment Dec 21 10:27:34.478430381 (378h58m34.349513416s) [upload=332595]: Waiting for error group managing segments... Dec 21 10:27:34.763620982 (378h58m34.634704027s) [noise=true , address=145.79.227.51:4411 , piece_id=LC6PCUVNJN77OICI5JGXWN7LJD6JGMKKHHIQG6T4MKX5SU7FGRIQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.770992478 (378h58m34.642075514s) [noise=true , address=145.79.227.50:4373 , piece_id=GVHEJYGQ6A3HKPZ4V563E37K4L3CZUDEGCVJJHGNC722L3BNY46Q, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.776530073 (378h58m34.647613108s) [noise=true , address=145.79.227.42:4121 , piece_id=FA7IZRDCG2XNN6PNDJOTZXGQXZWDOVV2XPR7MUSLK3YVYJVZE4UQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.786521177 (378h58m34.657604222s) [noise=true , address=145.79.227.17:4972 , piece_id=B3KYU5P3RL2BFR7QTS5ZK7B6UBDB3DTNTQKAGPM7B75S2SYEUFBQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.786591240 (378h58m34.657674275s) [noise=true , address=145.79.227.23:4547 , piece_id=OOD2VMVRTCVGANLHQJME4HCJACBJEH2OUSKZBUY6RVNMSXA3CM7Q, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.787618627 (378h58m34.658701672s) [noise=true , address=145.79.227.52:4445 , piece_id=YWZ6UHBVJWXLIGSE5HMV4WWZLLQLMBCB35WDSUFXK4XMBU5ZAT7Q, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.794650276 (378h58m34.665733311s) [noise=true , address=145.79.227.35:4482 , piece_id=3FUQ4OSF4B5DNWTAPY5W3LS2ZFBLCDWGXXLGASBKACSX3ONKLWWQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.794701083 (378h58m34.665784128s) [noise=true , address=145.79.227.39:4015 , piece_id=QB5S57M5HEYFEZOZYHBWMNFWVLQQEGHS73TBMANOJCCB42LK2AVA, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.833604161 (378h58m34.704687196s) [noise=true , address=145.79.227.48:4303 , piece_id=44PNGUZODJGAD7AFPZGJE6XBDYCSN2CLPIARGZI62PQ7KKU7ANWQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.857890877 (378h58m34.728973912s) [noise=true , address=145.79.227.27:4691 , piece_id=PEBFRJ6FC4PNFEODQJFGBHHC56RKLUECXWW36YBHNO47HDCDXJWQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.991414468 (378h58m34.862497503s) [noise=true , address=45.91.116.7:4011 , piece_id=UH4YHD7JFAKXOXJTNKRAUITFZVFWU5AJG5XHSLCJ6JS7JCLWEVRQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:35.048985192 (378h58m34.920068237s) [noise=true , address=45.91.116.14:4189 , piece_id=NF2SZ7V2LDV76ADGQ6NXVNBHTHNLMRYS4SXW6BT4OM7BQ5TQ6BPQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:35.055578377 (378h58m34.926661412s) [noise=true , address=45.91.116.8:4048 , piece_id=3G4VLJ76W4UGBDWZEIQCBDFPRGAM7QL4GMC23LS73BC3JSFGGHPA, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:35.055594557 (378h58m34.926677602s) [seg_pos={0 0}, upload=332595]: Segment reached optimal threshold of 13 pieces. Dec 21 10:27:35.055773338 (378h58m34.926856373s) [noise=true , address=45.91.116.4:4940 , piece_id=L6QW3XENG6UBHKUUXTMOQQRIXBWBN6XXL7SHLY7W5KC5ES3SFS3A, seg_pos={0 0}, upload=332595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 21 10:27:35.055779740 (378h58m34.926862775s) [noise=true , address=45.91.116.4:4940 , piece_id=L6QW3XENG6UBHKUUXTMOQQRIXBWBN6XXL7SHLY7W5KC5ES3SFS3A, seg_pos={0 0}, upload=332595]: Long tail cancellation due to optimal threshold Dec 21 10:27:35.055834805 (378h58m34.92691785s) [noise=true , address=77.90.31.15:4700 , piece_id=WQP7T5QHGW23C7I5TBY7PESM5YTTXS3BAEH3RDRQTJZYHI37BUJQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 21 10:27:35.055845194 (378h58m34.926928239s) [noise=true , address=77.90.31.15:4700 , piece_id=WQP7T5QHGW23C7I5TBY7PESM5YTTXS3BAEH3RDRQTJZYHI37BUJQ, seg_pos={0 0}, upload=332595]: Long tail cancellation due to optimal threshold Dec 21 10:27:35.055982826 (378h58m34.927065862s) [noise=true , address=45.91.116.12:4225 , piece_id=WH4P44CPP53IJXXMGK5VK2E3PYBGFLJBWSCFSCBKR2QTALIIEGEA, seg_pos={0 0}, upload=332595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 21 10:27:35.056041539 (378h58m34.927124594s) [noise=true , address=45.91.116.12:4225 , piece_id=WH4P44CPP53IJXXMGK5VK2E3PYBGFLJBWSCFSCBKR2QTALIIEGEA, seg_pos={0 0}, upload=332595]: Long tail cancellation due to optimal threshold Dec 21 10:27:35.056064683 (378h58m34.927147728s) [seg_pos={0 0}, upload=332595]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 10:27:35.056146969 (378h58m34.927230004s) [upload=332595]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 10:27:35.056153121 (378h58m34.927236176s) [upload=332595]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 10:27:35.353226207 (378h58m35.224309242s) [upload=332595]: Done uploading segments... Dec 21 10:27:35.353243942 (378h58m35.224326977s) [upload=332595]: Upload finished. err: <nil> Duration: 5.699902226s core uplink us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 729.820219ms trace: https://jaeger.dev.storj.io/trace/37011901b11802c9 FAIL (2025-12-21 10:46:36 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:37524->45.38.24.2:443: read: connection reset by peer Duration: 729.84741ms edge authservice http us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 5.691783877s trace: https://jaeger.dev.storj.io/trace/3ecb905698746e72 FAIL (2025-12-21 10:46:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: UH4YHD7JFAKXOXJTNKRAUITFZVFWU5AJG5XHSLCJ6JS7JCLWEVRQ) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 3G4VLJ76W4UGBDWZEIQCBDFPRGAM7QL4GMC23LS73BC3JSFGGHPA) context deadline exceeded ======LOGS====== Dec 21 10:27:32.520070050 (378h58m32.391153085s) [upload=332595]: Starting upload Dec 21 10:27:32.520134143 (378h58m32.391217178s) [upload=332595]: Done starting upload Dec 21 10:27:32.520194427 (378h58m32.391277472s) [upload=332595]: Uploading segments... Dec 21 10:27:32.520306451 (378h58m32.391389486s) [upload=332595]: Got next segment. Inline: false Dec 21 10:27:32.520335335 (378h58m32.39141839s) [upload=332595]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 10:27:32.520339694 (378h58m32.391422729s) [upload=332595]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 10:27:32.777396672 (378h58m32.648479718s) [seg_pos={0 0}, upload=332595]: Begin upload segment... Dec 21 10:27:32.777460795 (378h58m32.64854384s) [seg_pos={0 0}, upload=332595]: Done begin upload segment. Dec 21 10:27:32.777515899 (378h58m32.648598934s) [noise=true , address=145.79.227.23:4547 , piece_id=OOD2VMVRTCVGANLHQJME4HCJACBJEH2OUSKZBUY6RVNMSXA3CM7Q, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.777528564 (378h58m32.648611619s) [noise=true , address=145.79.227.27:4691 , piece_id=PEBFRJ6FC4PNFEODQJFGBHHC56RKLUECXWW36YBHNO47HDCDXJWQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.777605250 (378h58m32.648688295s) [noise=true , address=145.79.227.50:4373 , piece_id=GVHEJYGQ6A3HKPZ4V563E37K4L3CZUDEGCVJJHGNC722L3BNY46Q, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.778388482 (378h58m32.649471527s) [noise=true , address=145.79.227.39:4015 , piece_id=QB5S57M5HEYFEZOZYHBWMNFWVLQQEGHS73TBMANOJCCB42LK2AVA, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.778834370 (378h58m32.649917405s) [noise=true , address=45.91.116.12:4225 , piece_id=WH4P44CPP53IJXXMGK5VK2E3PYBGFLJBWSCFSCBKR2QTALIIEGEA, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.778858857 (378h58m32.649941892s) [noise=true , address=145.79.227.42:4121 , piece_id=FA7IZRDCG2XNN6PNDJOTZXGQXZWDOVV2XPR7MUSLK3YVYJVZE4UQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779235654 (378h58m32.650318689s) [noise=true , address=45.91.116.7:4011 , piece_id=UH4YHD7JFAKXOXJTNKRAUITFZVFWU5AJG5XHSLCJ6JS7JCLWEVRQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779269559 (378h58m32.650352594s) [noise=true , address=45.91.116.4:4940 , piece_id=L6QW3XENG6UBHKUUXTMOQQRIXBWBN6XXL7SHLY7W5KC5ES3SFS3A, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779536989 (378h58m32.650620024s) [noise=true , address=145.79.227.52:4445 , piece_id=YWZ6UHBVJWXLIGSE5HMV4WWZLLQLMBCB35WDSUFXK4XMBU5ZAT7Q, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779807954 (378h58m32.650890999s) [noise=true , address=77.90.31.15:4700 , piece_id=WQP7T5QHGW23C7I5TBY7PESM5YTTXS3BAEH3RDRQTJZYHI37BUJQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.780102345 (378h58m32.65118538s) [noise=true , address=145.79.227.17:4972 , piece_id=B3KYU5P3RL2BFR7QTS5ZK7B6UBDB3DTNTQKAGPM7B75S2SYEUFBQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.780482589 (378h58m32.651565634s) [noise=true , address=45.91.116.8:4048 , piece_id=3G4VLJ76W4UGBDWZEIQCBDFPRGAM7QL4GMC23LS73BC3JSFGGHPA, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.780863915 (378h58m32.65194696s) [noise=true , address=145.79.227.48:4303 , piece_id=44PNGUZODJGAD7AFPZGJE6XBDYCSN2CLPIARGZI62PQ7KKU7ANWQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.781783866 (378h58m32.652866911s) [noise=true , address=145.79.227.35:4482 , piece_id=3FUQ4OSF4B5DNWTAPY5W3LS2ZFBLCDWGXXLGASBKACSX3ONKLWWQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.782593317 (378h58m32.653676393s) [noise=true , address=45.91.116.14:4189 , piece_id=NF2SZ7V2LDV76ADGQ6NXVNBHTHNLMRYS4SXW6BT4OM7BQ5TQ6BPQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.783214190 (378h58m32.654297225s) [noise=true , address=145.79.227.51:4411 , piece_id=LC6PCUVNJN77OICI5JGXWN7LJD6JGMKKHHIQG6T4MKX5SU7FGRIQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:34.478418529 (378h58m34.349501564s) [upload=332595]: Next returned nil segment Dec 21 10:27:34.478430381 (378h58m34.349513416s) [upload=332595]: Waiting for error group managing segments... Dec 21 10:27:34.763620982 (378h58m34.634704027s) [noise=true , address=145.79.227.51:4411 , piece_id=LC6PCUVNJN77OICI5JGXWN7LJD6JGMKKHHIQG6T4MKX5SU7FGRIQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.770992478 (378h58m34.642075514s) [noise=true , address=145.79.227.50:4373 , piece_id=GVHEJYGQ6A3HKPZ4V563E37K4L3CZUDEGCVJJHGNC722L3BNY46Q, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.776530073 (378h58m34.647613108s) [noise=true , address=145.79.227.42:4121 , piece_id=FA7IZRDCG2XNN6PNDJOTZXGQXZWDOVV2XPR7MUSLK3YVYJVZE4UQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.786521177 (378h58m34.657604222s) [noise=true , address=145.79.227.17:4972 , piece_id=B3KYU5P3RL2BFR7QTS5ZK7B6UBDB3DTNTQKAGPM7B75S2SYEUFBQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.786591240 (378h58m34.657674275s) [noise=true , address=145.79.227.23:4547 , piece_id=OOD2VMVRTCVGANLHQJME4HCJACBJEH2OUSKZBUY6RVNMSXA3CM7Q, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.787618627 (378h58m34.658701672s) [noise=true , address=145.79.227.52:4445 , piece_id=YWZ6UHBVJWXLIGSE5HMV4WWZLLQLMBCB35WDSUFXK4XMBU5ZAT7Q, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.794650276 (378h58m34.665733311s) [noise=true , address=145.79.227.35:4482 , piece_id=3FUQ4OSF4B5DNWTAPY5W3LS2ZFBLCDWGXXLGASBKACSX3ONKLWWQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.794701083 (378h58m34.665784128s) [noise=true , address=145.79.227.39:4015 , piece_id=QB5S57M5HEYFEZOZYHBWMNFWVLQQEGHS73TBMANOJCCB42LK2AVA, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.833604161 (378h58m34.704687196s) [noise=true , address=145.79.227.48:4303 , piece_id=44PNGUZODJGAD7AFPZGJE6XBDYCSN2CLPIARGZI62PQ7KKU7ANWQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.857890877 (378h58m34.728973912s) [noise=true , address=145.79.227.27:4691 , piece_id=PEBFRJ6FC4PNFEODQJFGBHHC56RKLUECXWW36YBHNO47HDCDXJWQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.991414468 (378h58m34.862497503s) [noise=true , address=45.91.116.7:4011 , piece_id=UH4YHD7JFAKXOXJTNKRAUITFZVFWU5AJG5XHSLCJ6JS7JCLWEVRQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:35.048985192 (378h58m34.920068237s) [noise=true , address=45.91.116.14:4189 , piece_id=NF2SZ7V2LDV76ADGQ6NXVNBHTHNLMRYS4SXW6BT4OM7BQ5TQ6BPQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:35.055578377 (378h58m34.926661412s) [noise=true , address=45.91.116.8:4048 , piece_id=3G4VLJ76W4UGBDWZEIQCBDFPRGAM7QL4GMC23LS73BC3JSFGGHPA, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:35.055594557 (378h58m34.926677602s) [seg_pos={0 0}, upload=332595]: Segment reached optimal threshold of 13 pieces. Dec 21 10:27:35.055773338 (378h58m34.926856373s) [noise=true , address=45.91.116.4:4940 , piece_id=L6QW3XENG6UBHKUUXTMOQQRIXBWBN6XXL7SHLY7W5KC5ES3SFS3A, seg_pos={0 0}, upload=332595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 21 10:27:35.055779740 (378h58m34.926862775s) [noise=true , address=45.91.116.4:4940 , piece_id=L6QW3XENG6UBHKUUXTMOQQRIXBWBN6XXL7SHLY7W5KC5ES3SFS3A, seg_pos={0 0}, upload=332595]: Long tail cancellation due to optimal threshold Dec 21 10:27:35.055834805 (378h58m34.92691785s) [noise=true , address=77.90.31.15:4700 , piece_id=WQP7T5QHGW23C7I5TBY7PESM5YTTXS3BAEH3RDRQTJZYHI37BUJQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 21 10:27:35.055845194 (378h58m34.926928239s) [noise=true , address=77.90.31.15:4700 , piece_id=WQP7T5QHGW23C7I5TBY7PESM5YTTXS3BAEH3RDRQTJZYHI37BUJQ, seg_pos={0 0}, upload=332595]: Long tail cancellation due to optimal threshold Dec 21 10:27:35.055982826 (378h58m34.927065862s) [noise=true , address=45.91.116.12:4225 , piece_id=WH4P44CPP53IJXXMGK5VK2E3PYBGFLJBWSCFSCBKR2QTALIIEGEA, seg_pos={0 0}, upload=332595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 21 10:27:35.056041539 (378h58m34.927124594s) [noise=true , address=45.91.116.12:4225 , piece_id=WH4P44CPP53IJXXMGK5VK2E3PYBGFLJBWSCFSCBKR2QTALIIEGEA, seg_pos={0 0}, upload=332595]: Long tail cancellation due to optimal threshold Dec 21 10:27:35.056064683 (378h58m34.927147728s) [seg_pos={0 0}, upload=332595]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 10:27:35.056146969 (378h58m34.927230004s) [upload=332595]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 10:27:35.056153121 (378h58m34.927236176s) [upload=332595]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 10:27:35.353226207 (378h58m35.224309242s) [upload=332595]: Done uploading segments... Dec 21 10:27:35.353243942 (378h58m35.224326977s) [upload=332595]: Upload finished. err: <nil> Duration: 5.691817092s core uplink us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 717.480963ms trace: https://jaeger.dev.storj.io/trace/37011901b11802c9 FAIL (2025-12-21 10:46:10 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:37524->45.38.24.2:443: read: connection reset by peer Duration: 717.513665ms edge authservice http us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 5.701468145s trace: https://jaeger.dev.storj.io/trace/3ecb905698746e72 FAIL (2025-12-21 10:45:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: UH4YHD7JFAKXOXJTNKRAUITFZVFWU5AJG5XHSLCJ6JS7JCLWEVRQ) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 3G4VLJ76W4UGBDWZEIQCBDFPRGAM7QL4GMC23LS73BC3JSFGGHPA) context deadline exceeded ======LOGS====== Dec 21 10:27:32.520070050 (378h58m32.391153085s) [upload=332595]: Starting upload Dec 21 10:27:32.520134143 (378h58m32.391217178s) [upload=332595]: Done starting upload Dec 21 10:27:32.520194427 (378h58m32.391277472s) [upload=332595]: Uploading segments... Dec 21 10:27:32.520306451 (378h58m32.391389486s) [upload=332595]: Got next segment. Inline: false Dec 21 10:27:32.520335335 (378h58m32.39141839s) [upload=332595]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 10:27:32.520339694 (378h58m32.391422729s) [upload=332595]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 10:27:32.777396672 (378h58m32.648479718s) [seg_pos={0 0}, upload=332595]: Begin upload segment... Dec 21 10:27:32.777460795 (378h58m32.64854384s) [seg_pos={0 0}, upload=332595]: Done begin upload segment. Dec 21 10:27:32.777515899 (378h58m32.648598934s) [noise=true , address=145.79.227.23:4547 , piece_id=OOD2VMVRTCVGANLHQJME4HCJACBJEH2OUSKZBUY6RVNMSXA3CM7Q, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.777528564 (378h58m32.648611619s) [noise=true , address=145.79.227.27:4691 , piece_id=PEBFRJ6FC4PNFEODQJFGBHHC56RKLUECXWW36YBHNO47HDCDXJWQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.777605250 (378h58m32.648688295s) [noise=true , address=145.79.227.50:4373 , piece_id=GVHEJYGQ6A3HKPZ4V563E37K4L3CZUDEGCVJJHGNC722L3BNY46Q, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.778388482 (378h58m32.649471527s) [noise=true , address=145.79.227.39:4015 , piece_id=QB5S57M5HEYFEZOZYHBWMNFWVLQQEGHS73TBMANOJCCB42LK2AVA, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.778834370 (378h58m32.649917405s) [noise=true , address=45.91.116.12:4225 , piece_id=WH4P44CPP53IJXXMGK5VK2E3PYBGFLJBWSCFSCBKR2QTALIIEGEA, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.778858857 (378h58m32.649941892s) [noise=true , address=145.79.227.42:4121 , piece_id=FA7IZRDCG2XNN6PNDJOTZXGQXZWDOVV2XPR7MUSLK3YVYJVZE4UQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779235654 (378h58m32.650318689s) [noise=true , address=45.91.116.7:4011 , piece_id=UH4YHD7JFAKXOXJTNKRAUITFZVFWU5AJG5XHSLCJ6JS7JCLWEVRQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779269559 (378h58m32.650352594s) [noise=true , address=45.91.116.4:4940 , piece_id=L6QW3XENG6UBHKUUXTMOQQRIXBWBN6XXL7SHLY7W5KC5ES3SFS3A, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779536989 (378h58m32.650620024s) [noise=true , address=145.79.227.52:4445 , piece_id=YWZ6UHBVJWXLIGSE5HMV4WWZLLQLMBCB35WDSUFXK4XMBU5ZAT7Q, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779807954 (378h58m32.650890999s) [noise=true , address=77.90.31.15:4700 , piece_id=WQP7T5QHGW23C7I5TBY7PESM5YTTXS3BAEH3RDRQTJZYHI37BUJQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.780102345 (378h58m32.65118538s) [noise=true , address=145.79.227.17:4972 , piece_id=B3KYU5P3RL2BFR7QTS5ZK7B6UBDB3DTNTQKAGPM7B75S2SYEUFBQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.780482589 (378h58m32.651565634s) [noise=true , address=45.91.116.8:4048 , piece_id=3G4VLJ76W4UGBDWZEIQCBDFPRGAM7QL4GMC23LS73BC3JSFGGHPA, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.780863915 (378h58m32.65194696s) [noise=true , address=145.79.227.48:4303 , piece_id=44PNGUZODJGAD7AFPZGJE6XBDYCSN2CLPIARGZI62PQ7KKU7ANWQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.781783866 (378h58m32.652866911s) [noise=true , address=145.79.227.35:4482 , piece_id=3FUQ4OSF4B5DNWTAPY5W3LS2ZFBLCDWGXXLGASBKACSX3ONKLWWQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.782593317 (378h58m32.653676393s) [noise=true , address=45.91.116.14:4189 , piece_id=NF2SZ7V2LDV76ADGQ6NXVNBHTHNLMRYS4SXW6BT4OM7BQ5TQ6BPQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.783214190 (378h58m32.654297225s) [noise=true , address=145.79.227.51:4411 , piece_id=LC6PCUVNJN77OICI5JGXWN7LJD6JGMKKHHIQG6T4MKX5SU7FGRIQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:34.478418529 (378h58m34.349501564s) [upload=332595]: Next returned nil segment Dec 21 10:27:34.478430381 (378h58m34.349513416s) [upload=332595]: Waiting for error group managing segments... Dec 21 10:27:34.763620982 (378h58m34.634704027s) [noise=true , address=145.79.227.51:4411 , piece_id=LC6PCUVNJN77OICI5JGXWN7LJD6JGMKKHHIQG6T4MKX5SU7FGRIQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.770992478 (378h58m34.642075514s) [noise=true , address=145.79.227.50:4373 , piece_id=GVHEJYGQ6A3HKPZ4V563E37K4L3CZUDEGCVJJHGNC722L3BNY46Q, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.776530073 (378h58m34.647613108s) [noise=true , address=145.79.227.42:4121 , piece_id=FA7IZRDCG2XNN6PNDJOTZXGQXZWDOVV2XPR7MUSLK3YVYJVZE4UQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.786521177 (378h58m34.657604222s) [noise=true , address=145.79.227.17:4972 , piece_id=B3KYU5P3RL2BFR7QTS5ZK7B6UBDB3DTNTQKAGPM7B75S2SYEUFBQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.786591240 (378h58m34.657674275s) [noise=true , address=145.79.227.23:4547 , piece_id=OOD2VMVRTCVGANLHQJME4HCJACBJEH2OUSKZBUY6RVNMSXA3CM7Q, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.787618627 (378h58m34.658701672s) [noise=true , address=145.79.227.52:4445 , piece_id=YWZ6UHBVJWXLIGSE5HMV4WWZLLQLMBCB35WDSUFXK4XMBU5ZAT7Q, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.794650276 (378h58m34.665733311s) [noise=true , address=145.79.227.35:4482 , piece_id=3FUQ4OSF4B5DNWTAPY5W3LS2ZFBLCDWGXXLGASBKACSX3ONKLWWQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.794701083 (378h58m34.665784128s) [noise=true , address=145.79.227.39:4015 , piece_id=QB5S57M5HEYFEZOZYHBWMNFWVLQQEGHS73TBMANOJCCB42LK2AVA, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.833604161 (378h58m34.704687196s) [noise=true , address=145.79.227.48:4303 , piece_id=44PNGUZODJGAD7AFPZGJE6XBDYCSN2CLPIARGZI62PQ7KKU7ANWQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.857890877 (378h58m34.728973912s) [noise=true , address=145.79.227.27:4691 , piece_id=PEBFRJ6FC4PNFEODQJFGBHHC56RKLUECXWW36YBHNO47HDCDXJWQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.991414468 (378h58m34.862497503s) [noise=true , address=45.91.116.7:4011 , piece_id=UH4YHD7JFAKXOXJTNKRAUITFZVFWU5AJG5XHSLCJ6JS7JCLWEVRQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:35.048985192 (378h58m34.920068237s) [noise=true , address=45.91.116.14:4189 , piece_id=NF2SZ7V2LDV76ADGQ6NXVNBHTHNLMRYS4SXW6BT4OM7BQ5TQ6BPQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:35.055578377 (378h58m34.926661412s) [noise=true , address=45.91.116.8:4048 , piece_id=3G4VLJ76W4UGBDWZEIQCBDFPRGAM7QL4GMC23LS73BC3JSFGGHPA, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:35.055594557 (378h58m34.926677602s) [seg_pos={0 0}, upload=332595]: Segment reached optimal threshold of 13 pieces. Dec 21 10:27:35.055773338 (378h58m34.926856373s) [noise=true , address=45.91.116.4:4940 , piece_id=L6QW3XENG6UBHKUUXTMOQQRIXBWBN6XXL7SHLY7W5KC5ES3SFS3A, seg_pos={0 0}, upload=332595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 21 10:27:35.055779740 (378h58m34.926862775s) [noise=true , address=45.91.116.4:4940 , piece_id=L6QW3XENG6UBHKUUXTMOQQRIXBWBN6XXL7SHLY7W5KC5ES3SFS3A, seg_pos={0 0}, upload=332595]: Long tail cancellation due to optimal threshold Dec 21 10:27:35.055834805 (378h58m34.92691785s) [noise=true , address=77.90.31.15:4700 , piece_id=WQP7T5QHGW23C7I5TBY7PESM5YTTXS3BAEH3RDRQTJZYHI37BUJQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 21 10:27:35.055845194 (378h58m34.926928239s) [noise=true , address=77.90.31.15:4700 , piece_id=WQP7T5QHGW23C7I5TBY7PESM5YTTXS3BAEH3RDRQTJZYHI37BUJQ, seg_pos={0 0}, upload=332595]: Long tail cancellation due to optimal threshold Dec 21 10:27:35.055982826 (378h58m34.927065862s) [noise=true , address=45.91.116.12:4225 , piece_id=WH4P44CPP53IJXXMGK5VK2E3PYBGFLJBWSCFSCBKR2QTALIIEGEA, seg_pos={0 0}, upload=332595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 21 10:27:35.056041539 (378h58m34.927124594s) [noise=true , address=45.91.116.12:4225 , piece_id=WH4P44CPP53IJXXMGK5VK2E3PYBGFLJBWSCFSCBKR2QTALIIEGEA, seg_pos={0 0}, upload=332595]: Long tail cancellation due to optimal threshold Dec 21 10:27:35.056064683 (378h58m34.927147728s) [seg_pos={0 0}, upload=332595]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 10:27:35.056146969 (378h58m34.927230004s) [upload=332595]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 10:27:35.056153121 (378h58m34.927236176s) [upload=332595]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 10:27:35.353226207 (378h58m35.224309242s) [upload=332595]: Done uploading segments... Dec 21 10:27:35.353243942 (378h58m35.224326977s) [upload=332595]: Upload finished. err: <nil> Duration: 5.701513521s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 9.494293783s trace: https://jaeger.dev.storj.io/trace/3ecb905698746e72 FAIL (2025-12-21 10:45:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: UH4YHD7JFAKXOXJTNKRAUITFZVFWU5AJG5XHSLCJ6JS7JCLWEVRQ) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 3G4VLJ76W4UGBDWZEIQCBDFPRGAM7QL4GMC23LS73BC3JSFGGHPA) context deadline exceeded ======LOGS====== Dec 21 10:27:32.520070050 (378h58m32.391153085s) [upload=332595]: Starting upload Dec 21 10:27:32.520134143 (378h58m32.391217178s) [upload=332595]: Done starting upload Dec 21 10:27:32.520194427 (378h58m32.391277472s) [upload=332595]: Uploading segments... Dec 21 10:27:32.520306451 (378h58m32.391389486s) [upload=332595]: Got next segment. Inline: false Dec 21 10:27:32.520335335 (378h58m32.39141839s) [upload=332595]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 10:27:32.520339694 (378h58m32.391422729s) [upload=332595]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 10:27:32.777396672 (378h58m32.648479718s) [seg_pos={0 0}, upload=332595]: Begin upload segment... Dec 21 10:27:32.777460795 (378h58m32.64854384s) [seg_pos={0 0}, upload=332595]: Done begin upload segment. Dec 21 10:27:32.777515899 (378h58m32.648598934s) [noise=true , address=145.79.227.23:4547 , piece_id=OOD2VMVRTCVGANLHQJME4HCJACBJEH2OUSKZBUY6RVNMSXA3CM7Q, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.777528564 (378h58m32.648611619s) [noise=true , address=145.79.227.27:4691 , piece_id=PEBFRJ6FC4PNFEODQJFGBHHC56RKLUECXWW36YBHNO47HDCDXJWQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.777605250 (378h58m32.648688295s) [noise=true , address=145.79.227.50:4373 , piece_id=GVHEJYGQ6A3HKPZ4V563E37K4L3CZUDEGCVJJHGNC722L3BNY46Q, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.778388482 (378h58m32.649471527s) [noise=true , address=145.79.227.39:4015 , piece_id=QB5S57M5HEYFEZOZYHBWMNFWVLQQEGHS73TBMANOJCCB42LK2AVA, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.778834370 (378h58m32.649917405s) [noise=true , address=45.91.116.12:4225 , piece_id=WH4P44CPP53IJXXMGK5VK2E3PYBGFLJBWSCFSCBKR2QTALIIEGEA, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.778858857 (378h58m32.649941892s) [noise=true , address=145.79.227.42:4121 , piece_id=FA7IZRDCG2XNN6PNDJOTZXGQXZWDOVV2XPR7MUSLK3YVYJVZE4UQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779235654 (378h58m32.650318689s) [noise=true , address=45.91.116.7:4011 , piece_id=UH4YHD7JFAKXOXJTNKRAUITFZVFWU5AJG5XHSLCJ6JS7JCLWEVRQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779269559 (378h58m32.650352594s) [noise=true , address=45.91.116.4:4940 , piece_id=L6QW3XENG6UBHKUUXTMOQQRIXBWBN6XXL7SHLY7W5KC5ES3SFS3A, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779536989 (378h58m32.650620024s) [noise=true , address=145.79.227.52:4445 , piece_id=YWZ6UHBVJWXLIGSE5HMV4WWZLLQLMBCB35WDSUFXK4XMBU5ZAT7Q, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779807954 (378h58m32.650890999s) [noise=true , address=77.90.31.15:4700 , piece_id=WQP7T5QHGW23C7I5TBY7PESM5YTTXS3BAEH3RDRQTJZYHI37BUJQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.780102345 (378h58m32.65118538s) [noise=true , address=145.79.227.17:4972 , piece_id=B3KYU5P3RL2BFR7QTS5ZK7B6UBDB3DTNTQKAGPM7B75S2SYEUFBQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.780482589 (378h58m32.651565634s) [noise=true , address=45.91.116.8:4048 , piece_id=3G4VLJ76W4UGBDWZEIQCBDFPRGAM7QL4GMC23LS73BC3JSFGGHPA, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.780863915 (378h58m32.65194696s) [noise=true , address=145.79.227.48:4303 , piece_id=44PNGUZODJGAD7AFPZGJE6XBDYCSN2CLPIARGZI62PQ7KKU7ANWQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.781783866 (378h58m32.652866911s) [noise=true , address=145.79.227.35:4482 , piece_id=3FUQ4OSF4B5DNWTAPY5W3LS2ZFBLCDWGXXLGASBKACSX3ONKLWWQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.782593317 (378h58m32.653676393s) [noise=true , address=45.91.116.14:4189 , piece_id=NF2SZ7V2LDV76ADGQ6NXVNBHTHNLMRYS4SXW6BT4OM7BQ5TQ6BPQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.783214190 (378h58m32.654297225s) [noise=true , address=145.79.227.51:4411 , piece_id=LC6PCUVNJN77OICI5JGXWN7LJD6JGMKKHHIQG6T4MKX5SU7FGRIQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:34.478418529 (378h58m34.349501564s) [upload=332595]: Next returned nil segment Dec 21 10:27:34.478430381 (378h58m34.349513416s) [upload=332595]: Waiting for error group managing segments... Dec 21 10:27:34.763620982 (378h58m34.634704027s) [noise=true , address=145.79.227.51:4411 , piece_id=LC6PCUVNJN77OICI5JGXWN7LJD6JGMKKHHIQG6T4MKX5SU7FGRIQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.770992478 (378h58m34.642075514s) [noise=true , address=145.79.227.50:4373 , piece_id=GVHEJYGQ6A3HKPZ4V563E37K4L3CZUDEGCVJJHGNC722L3BNY46Q, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.776530073 (378h58m34.647613108s) [noise=true , address=145.79.227.42:4121 , piece_id=FA7IZRDCG2XNN6PNDJOTZXGQXZWDOVV2XPR7MUSLK3YVYJVZE4UQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.786521177 (378h58m34.657604222s) [noise=true , address=145.79.227.17:4972 , piece_id=B3KYU5P3RL2BFR7QTS5ZK7B6UBDB3DTNTQKAGPM7B75S2SYEUFBQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.786591240 (378h58m34.657674275s) [noise=true , address=145.79.227.23:4547 , piece_id=OOD2VMVRTCVGANLHQJME4HCJACBJEH2OUSKZBUY6RVNMSXA3CM7Q, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.787618627 (378h58m34.658701672s) [noise=true , address=145.79.227.52:4445 , piece_id=YWZ6UHBVJWXLIGSE5HMV4WWZLLQLMBCB35WDSUFXK4XMBU5ZAT7Q, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.794650276 (378h58m34.665733311s) [noise=true , address=145.79.227.35:4482 , piece_id=3FUQ4OSF4B5DNWTAPY5W3LS2ZFBLCDWGXXLGASBKACSX3ONKLWWQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.794701083 (378h58m34.665784128s) [noise=true , address=145.79.227.39:4015 , piece_id=QB5S57M5HEYFEZOZYHBWMNFWVLQQEGHS73TBMANOJCCB42LK2AVA, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.833604161 (378h58m34.704687196s) [noise=true , address=145.79.227.48:4303 , piece_id=44PNGUZODJGAD7AFPZGJE6XBDYCSN2CLPIARGZI62PQ7KKU7ANWQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.857890877 (378h58m34.728973912s) [noise=true , address=145.79.227.27:4691 , piece_id=PEBFRJ6FC4PNFEODQJFGBHHC56RKLUECXWW36YBHNO47HDCDXJWQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.991414468 (378h58m34.862497503s) [noise=true , address=45.91.116.7:4011 , piece_id=UH4YHD7JFAKXOXJTNKRAUITFZVFWU5AJG5XHSLCJ6JS7JCLWEVRQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:35.048985192 (378h58m34.920068237s) [noise=true , address=45.91.116.14:4189 , piece_id=NF2SZ7V2LDV76ADGQ6NXVNBHTHNLMRYS4SXW6BT4OM7BQ5TQ6BPQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:35.055578377 (378h58m34.926661412s) [noise=true , address=45.91.116.8:4048 , piece_id=3G4VLJ76W4UGBDWZEIQCBDFPRGAM7QL4GMC23LS73BC3JSFGGHPA, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:35.055594557 (378h58m34.926677602s) [seg_pos={0 0}, upload=332595]: Segment reached optimal threshold of 13 pieces. Dec 21 10:27:35.055773338 (378h58m34.926856373s) [noise=true , address=45.91.116.4:4940 , piece_id=L6QW3XENG6UBHKUUXTMOQQRIXBWBN6XXL7SHLY7W5KC5ES3SFS3A, seg_pos={0 0}, upload=332595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 21 10:27:35.055779740 (378h58m34.926862775s) [noise=true , address=45.91.116.4:4940 , piece_id=L6QW3XENG6UBHKUUXTMOQQRIXBWBN6XXL7SHLY7W5KC5ES3SFS3A, seg_pos={0 0}, upload=332595]: Long tail cancellation due to optimal threshold Dec 21 10:27:35.055834805 (378h58m34.92691785s) [noise=true , address=77.90.31.15:4700 , piece_id=WQP7T5QHGW23C7I5TBY7PESM5YTTXS3BAEH3RDRQTJZYHI37BUJQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 21 10:27:35.055845194 (378h58m34.926928239s) [noise=true , address=77.90.31.15:4700 , piece_id=WQP7T5QHGW23C7I5TBY7PESM5YTTXS3BAEH3RDRQTJZYHI37BUJQ, seg_pos={0 0}, upload=332595]: Long tail cancellation due to optimal threshold Dec 21 10:27:35.055982826 (378h58m34.927065862s) [noise=true , address=45.91.116.12:4225 , piece_id=WH4P44CPP53IJXXMGK5VK2E3PYBGFLJBWSCFSCBKR2QTALIIEGEA, seg_pos={0 0}, upload=332595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 21 10:27:35.056041539 (378h58m34.927124594s) [noise=true , address=45.91.116.12:4225 , piece_id=WH4P44CPP53IJXXMGK5VK2E3PYBGFLJBWSCFSCBKR2QTALIIEGEA, seg_pos={0 0}, upload=332595]: Long tail cancellation due to optimal threshold Dec 21 10:27:35.056064683 (378h58m34.927147728s) [seg_pos={0 0}, upload=332595]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 10:27:35.056146969 (378h58m34.927230004s) [upload=332595]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 10:27:35.056153121 (378h58m34.927236176s) [upload=332595]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 10:27:35.353226207 (378h58m35.224309242s) [upload=332595]: Done uploading segments... Dec 21 10:27:35.353243942 (378h58m35.224326977s) [upload=332595]: Upload finished. err: <nil> Duration: 9.494325394s core uplink us-select-1
checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 684.721303ms trace: https://jaeger.dev.storj.io/trace/37011901b11802c9 FAIL (2025-12-21 10:45:16 UTC) orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": read tcp 188.166.225.170:37524->45.38.24.2:443: read: connection reset by peer Duration: 684.744647ms edge authservice http us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 5.735660998s trace: https://jaeger.dev.storj.io/trace/3ecb905698746e72 FAIL (2025-12-21 10:44:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: UH4YHD7JFAKXOXJTNKRAUITFZVFWU5AJG5XHSLCJ6JS7JCLWEVRQ) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 3G4VLJ76W4UGBDWZEIQCBDFPRGAM7QL4GMC23LS73BC3JSFGGHPA) context deadline exceeded ======LOGS====== Dec 21 10:27:32.520070050 (378h58m32.391153085s) [upload=332595]: Starting upload Dec 21 10:27:32.520134143 (378h58m32.391217178s) [upload=332595]: Done starting upload Dec 21 10:27:32.520194427 (378h58m32.391277472s) [upload=332595]: Uploading segments... Dec 21 10:27:32.520306451 (378h58m32.391389486s) [upload=332595]: Got next segment. Inline: false Dec 21 10:27:32.520335335 (378h58m32.39141839s) [upload=332595]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 10:27:32.520339694 (378h58m32.391422729s) [upload=332595]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 10:27:32.777396672 (378h58m32.648479718s) [seg_pos={0 0}, upload=332595]: Begin upload segment... Dec 21 10:27:32.777460795 (378h58m32.64854384s) [seg_pos={0 0}, upload=332595]: Done begin upload segment. Dec 21 10:27:32.777515899 (378h58m32.648598934s) [noise=true , address=145.79.227.23:4547 , piece_id=OOD2VMVRTCVGANLHQJME4HCJACBJEH2OUSKZBUY6RVNMSXA3CM7Q, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.777528564 (378h58m32.648611619s) [noise=true , address=145.79.227.27:4691 , piece_id=PEBFRJ6FC4PNFEODQJFGBHHC56RKLUECXWW36YBHNO47HDCDXJWQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.777605250 (378h58m32.648688295s) [noise=true , address=145.79.227.50:4373 , piece_id=GVHEJYGQ6A3HKPZ4V563E37K4L3CZUDEGCVJJHGNC722L3BNY46Q, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.778388482 (378h58m32.649471527s) [noise=true , address=145.79.227.39:4015 , piece_id=QB5S57M5HEYFEZOZYHBWMNFWVLQQEGHS73TBMANOJCCB42LK2AVA, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.778834370 (378h58m32.649917405s) [noise=true , address=45.91.116.12:4225 , piece_id=WH4P44CPP53IJXXMGK5VK2E3PYBGFLJBWSCFSCBKR2QTALIIEGEA, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.778858857 (378h58m32.649941892s) [noise=true , address=145.79.227.42:4121 , piece_id=FA7IZRDCG2XNN6PNDJOTZXGQXZWDOVV2XPR7MUSLK3YVYJVZE4UQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779235654 (378h58m32.650318689s) [noise=true , address=45.91.116.7:4011 , piece_id=UH4YHD7JFAKXOXJTNKRAUITFZVFWU5AJG5XHSLCJ6JS7JCLWEVRQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779269559 (378h58m32.650352594s) [noise=true , address=45.91.116.4:4940 , piece_id=L6QW3XENG6UBHKUUXTMOQQRIXBWBN6XXL7SHLY7W5KC5ES3SFS3A, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779536989 (378h58m32.650620024s) [noise=true , address=145.79.227.52:4445 , piece_id=YWZ6UHBVJWXLIGSE5HMV4WWZLLQLMBCB35WDSUFXK4XMBU5ZAT7Q, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779807954 (378h58m32.650890999s) [noise=true , address=77.90.31.15:4700 , piece_id=WQP7T5QHGW23C7I5TBY7PESM5YTTXS3BAEH3RDRQTJZYHI37BUJQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.780102345 (378h58m32.65118538s) [noise=true , address=145.79.227.17:4972 , piece_id=B3KYU5P3RL2BFR7QTS5ZK7B6UBDB3DTNTQKAGPM7B75S2SYEUFBQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.780482589 (378h58m32.651565634s) [noise=true , address=45.91.116.8:4048 , piece_id=3G4VLJ76W4UGBDWZEIQCBDFPRGAM7QL4GMC23LS73BC3JSFGGHPA, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.780863915 (378h58m32.65194696s) [noise=true , address=145.79.227.48:4303 , piece_id=44PNGUZODJGAD7AFPZGJE6XBDYCSN2CLPIARGZI62PQ7KKU7ANWQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.781783866 (378h58m32.652866911s) [noise=true , address=145.79.227.35:4482 , piece_id=3FUQ4OSF4B5DNWTAPY5W3LS2ZFBLCDWGXXLGASBKACSX3ONKLWWQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.782593317 (378h58m32.653676393s) [noise=true , address=45.91.116.14:4189 , piece_id=NF2SZ7V2LDV76ADGQ6NXVNBHTHNLMRYS4SXW6BT4OM7BQ5TQ6BPQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.783214190 (378h58m32.654297225s) [noise=true , address=145.79.227.51:4411 , piece_id=LC6PCUVNJN77OICI5JGXWN7LJD6JGMKKHHIQG6T4MKX5SU7FGRIQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:34.478418529 (378h58m34.349501564s) [upload=332595]: Next returned nil segment Dec 21 10:27:34.478430381 (378h58m34.349513416s) [upload=332595]: Waiting for error group managing segments... Dec 21 10:27:34.763620982 (378h58m34.634704027s) [noise=true , address=145.79.227.51:4411 , piece_id=LC6PCUVNJN77OICI5JGXWN7LJD6JGMKKHHIQG6T4MKX5SU7FGRIQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.770992478 (378h58m34.642075514s) [noise=true , address=145.79.227.50:4373 , piece_id=GVHEJYGQ6A3HKPZ4V563E37K4L3CZUDEGCVJJHGNC722L3BNY46Q, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.776530073 (378h58m34.647613108s) [noise=true , address=145.79.227.42:4121 , piece_id=FA7IZRDCG2XNN6PNDJOTZXGQXZWDOVV2XPR7MUSLK3YVYJVZE4UQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.786521177 (378h58m34.657604222s) [noise=true , address=145.79.227.17:4972 , piece_id=B3KYU5P3RL2BFR7QTS5ZK7B6UBDB3DTNTQKAGPM7B75S2SYEUFBQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.786591240 (378h58m34.657674275s) [noise=true , address=145.79.227.23:4547 , piece_id=OOD2VMVRTCVGANLHQJME4HCJACBJEH2OUSKZBUY6RVNMSXA3CM7Q, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.787618627 (378h58m34.658701672s) [noise=true , address=145.79.227.52:4445 , piece_id=YWZ6UHBVJWXLIGSE5HMV4WWZLLQLMBCB35WDSUFXK4XMBU5ZAT7Q, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.794650276 (378h58m34.665733311s) [noise=true , address=145.79.227.35:4482 , piece_id=3FUQ4OSF4B5DNWTAPY5W3LS2ZFBLCDWGXXLGASBKACSX3ONKLWWQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.794701083 (378h58m34.665784128s) [noise=true , address=145.79.227.39:4015 , piece_id=QB5S57M5HEYFEZOZYHBWMNFWVLQQEGHS73TBMANOJCCB42LK2AVA, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.833604161 (378h58m34.704687196s) [noise=true , address=145.79.227.48:4303 , piece_id=44PNGUZODJGAD7AFPZGJE6XBDYCSN2CLPIARGZI62PQ7KKU7ANWQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.857890877 (378h58m34.728973912s) [noise=true , address=145.79.227.27:4691 , piece_id=PEBFRJ6FC4PNFEODQJFGBHHC56RKLUECXWW36YBHNO47HDCDXJWQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.991414468 (378h58m34.862497503s) [noise=true , address=45.91.116.7:4011 , piece_id=UH4YHD7JFAKXOXJTNKRAUITFZVFWU5AJG5XHSLCJ6JS7JCLWEVRQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:35.048985192 (378h58m34.920068237s) [noise=true , address=45.91.116.14:4189 , piece_id=NF2SZ7V2LDV76ADGQ6NXVNBHTHNLMRYS4SXW6BT4OM7BQ5TQ6BPQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:35.055578377 (378h58m34.926661412s) [noise=true , address=45.91.116.8:4048 , piece_id=3G4VLJ76W4UGBDWZEIQCBDFPRGAM7QL4GMC23LS73BC3JSFGGHPA, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:35.055594557 (378h58m34.926677602s) [seg_pos={0 0}, upload=332595]: Segment reached optimal threshold of 13 pieces. Dec 21 10:27:35.055773338 (378h58m34.926856373s) [noise=true , address=45.91.116.4:4940 , piece_id=L6QW3XENG6UBHKUUXTMOQQRIXBWBN6XXL7SHLY7W5KC5ES3SFS3A, seg_pos={0 0}, upload=332595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 21 10:27:35.055779740 (378h58m34.926862775s) [noise=true , address=45.91.116.4:4940 , piece_id=L6QW3XENG6UBHKUUXTMOQQRIXBWBN6XXL7SHLY7W5KC5ES3SFS3A, seg_pos={0 0}, upload=332595]: Long tail cancellation due to optimal threshold Dec 21 10:27:35.055834805 (378h58m34.92691785s) [noise=true , address=77.90.31.15:4700 , piece_id=WQP7T5QHGW23C7I5TBY7PESM5YTTXS3BAEH3RDRQTJZYHI37BUJQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 21 10:27:35.055845194 (378h58m34.926928239s) [noise=true , address=77.90.31.15:4700 , piece_id=WQP7T5QHGW23C7I5TBY7PESM5YTTXS3BAEH3RDRQTJZYHI37BUJQ, seg_pos={0 0}, upload=332595]: Long tail cancellation due to optimal threshold Dec 21 10:27:35.055982826 (378h58m34.927065862s) [noise=true , address=45.91.116.12:4225 , piece_id=WH4P44CPP53IJXXMGK5VK2E3PYBGFLJBWSCFSCBKR2QTALIIEGEA, seg_pos={0 0}, upload=332595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 21 10:27:35.056041539 (378h58m34.927124594s) [noise=true , address=45.91.116.12:4225 , piece_id=WH4P44CPP53IJXXMGK5VK2E3PYBGFLJBWSCFSCBKR2QTALIIEGEA, seg_pos={0 0}, upload=332595]: Long tail cancellation due to optimal threshold Dec 21 10:27:35.056064683 (378h58m34.927147728s) [seg_pos={0 0}, upload=332595]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 10:27:35.056146969 (378h58m34.927230004s) [upload=332595]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 10:27:35.056153121 (378h58m34.927236176s) [upload=332595]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 10:27:35.353226207 (378h58m35.224309242s) [upload=332595]: Done uploading segments... Dec 21 10:27:35.353243942 (378h58m35.224326977s) [upload=332595]: Upload finished. err: <nil> Duration: 5.735691406s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 5.892321593s trace: https://jaeger.dev.storj.io/trace/3ecb905698746e72 FAIL (2025-12-21 10:43:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: UH4YHD7JFAKXOXJTNKRAUITFZVFWU5AJG5XHSLCJ6JS7JCLWEVRQ) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 3G4VLJ76W4UGBDWZEIQCBDFPRGAM7QL4GMC23LS73BC3JSFGGHPA) context deadline exceeded ======LOGS====== Dec 21 10:27:32.520070050 (378h58m32.391153085s) [upload=332595]: Starting upload Dec 21 10:27:32.520134143 (378h58m32.391217178s) [upload=332595]: Done starting upload Dec 21 10:27:32.520194427 (378h58m32.391277472s) [upload=332595]: Uploading segments... Dec 21 10:27:32.520306451 (378h58m32.391389486s) [upload=332595]: Got next segment. Inline: false Dec 21 10:27:32.520335335 (378h58m32.39141839s) [upload=332595]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 10:27:32.520339694 (378h58m32.391422729s) [upload=332595]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 10:27:32.777396672 (378h58m32.648479718s) [seg_pos={0 0}, upload=332595]: Begin upload segment... Dec 21 10:27:32.777460795 (378h58m32.64854384s) [seg_pos={0 0}, upload=332595]: Done begin upload segment. Dec 21 10:27:32.777515899 (378h58m32.648598934s) [noise=true , address=145.79.227.23:4547 , piece_id=OOD2VMVRTCVGANLHQJME4HCJACBJEH2OUSKZBUY6RVNMSXA3CM7Q, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.777528564 (378h58m32.648611619s) [noise=true , address=145.79.227.27:4691 , piece_id=PEBFRJ6FC4PNFEODQJFGBHHC56RKLUECXWW36YBHNO47HDCDXJWQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.777605250 (378h58m32.648688295s) [noise=true , address=145.79.227.50:4373 , piece_id=GVHEJYGQ6A3HKPZ4V563E37K4L3CZUDEGCVJJHGNC722L3BNY46Q, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.778388482 (378h58m32.649471527s) [noise=true , address=145.79.227.39:4015 , piece_id=QB5S57M5HEYFEZOZYHBWMNFWVLQQEGHS73TBMANOJCCB42LK2AVA, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.778834370 (378h58m32.649917405s) [noise=true , address=45.91.116.12:4225 , piece_id=WH4P44CPP53IJXXMGK5VK2E3PYBGFLJBWSCFSCBKR2QTALIIEGEA, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.778858857 (378h58m32.649941892s) [noise=true , address=145.79.227.42:4121 , piece_id=FA7IZRDCG2XNN6PNDJOTZXGQXZWDOVV2XPR7MUSLK3YVYJVZE4UQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779235654 (378h58m32.650318689s) [noise=true , address=45.91.116.7:4011 , piece_id=UH4YHD7JFAKXOXJTNKRAUITFZVFWU5AJG5XHSLCJ6JS7JCLWEVRQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779269559 (378h58m32.650352594s) [noise=true , address=45.91.116.4:4940 , piece_id=L6QW3XENG6UBHKUUXTMOQQRIXBWBN6XXL7SHLY7W5KC5ES3SFS3A, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779536989 (378h58m32.650620024s) [noise=true , address=145.79.227.52:4445 , piece_id=YWZ6UHBVJWXLIGSE5HMV4WWZLLQLMBCB35WDSUFXK4XMBU5ZAT7Q, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779807954 (378h58m32.650890999s) [noise=true , address=77.90.31.15:4700 , piece_id=WQP7T5QHGW23C7I5TBY7PESM5YTTXS3BAEH3RDRQTJZYHI37BUJQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.780102345 (378h58m32.65118538s) [noise=true , address=145.79.227.17:4972 , piece_id=B3KYU5P3RL2BFR7QTS5ZK7B6UBDB3DTNTQKAGPM7B75S2SYEUFBQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.780482589 (378h58m32.651565634s) [noise=true , address=45.91.116.8:4048 , piece_id=3G4VLJ76W4UGBDWZEIQCBDFPRGAM7QL4GMC23LS73BC3JSFGGHPA, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.780863915 (378h58m32.65194696s) [noise=true , address=145.79.227.48:4303 , piece_id=44PNGUZODJGAD7AFPZGJE6XBDYCSN2CLPIARGZI62PQ7KKU7ANWQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.781783866 (378h58m32.652866911s) [noise=true , address=145.79.227.35:4482 , piece_id=3FUQ4OSF4B5DNWTAPY5W3LS2ZFBLCDWGXXLGASBKACSX3ONKLWWQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.782593317 (378h58m32.653676393s) [noise=true , address=45.91.116.14:4189 , piece_id=NF2SZ7V2LDV76ADGQ6NXVNBHTHNLMRYS4SXW6BT4OM7BQ5TQ6BPQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.783214190 (378h58m32.654297225s) [noise=true , address=145.79.227.51:4411 , piece_id=LC6PCUVNJN77OICI5JGXWN7LJD6JGMKKHHIQG6T4MKX5SU7FGRIQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:34.478418529 (378h58m34.349501564s) [upload=332595]: Next returned nil segment Dec 21 10:27:34.478430381 (378h58m34.349513416s) [upload=332595]: Waiting for error group managing segments... Dec 21 10:27:34.763620982 (378h58m34.634704027s) [noise=true , address=145.79.227.51:4411 , piece_id=LC6PCUVNJN77OICI5JGXWN7LJD6JGMKKHHIQG6T4MKX5SU7FGRIQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.770992478 (378h58m34.642075514s) [noise=true , address=145.79.227.50:4373 , piece_id=GVHEJYGQ6A3HKPZ4V563E37K4L3CZUDEGCVJJHGNC722L3BNY46Q, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.776530073 (378h58m34.647613108s) [noise=true , address=145.79.227.42:4121 , piece_id=FA7IZRDCG2XNN6PNDJOTZXGQXZWDOVV2XPR7MUSLK3YVYJVZE4UQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.786521177 (378h58m34.657604222s) [noise=true , address=145.79.227.17:4972 , piece_id=B3KYU5P3RL2BFR7QTS5ZK7B6UBDB3DTNTQKAGPM7B75S2SYEUFBQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.786591240 (378h58m34.657674275s) [noise=true , address=145.79.227.23:4547 , piece_id=OOD2VMVRTCVGANLHQJME4HCJACBJEH2OUSKZBUY6RVNMSXA3CM7Q, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.787618627 (378h58m34.658701672s) [noise=true , address=145.79.227.52:4445 , piece_id=YWZ6UHBVJWXLIGSE5HMV4WWZLLQLMBCB35WDSUFXK4XMBU5ZAT7Q, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.794650276 (378h58m34.665733311s) [noise=true , address=145.79.227.35:4482 , piece_id=3FUQ4OSF4B5DNWTAPY5W3LS2ZFBLCDWGXXLGASBKACSX3ONKLWWQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.794701083 (378h58m34.665784128s) [noise=true , address=145.79.227.39:4015 , piece_id=QB5S57M5HEYFEZOZYHBWMNFWVLQQEGHS73TBMANOJCCB42LK2AVA, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.833604161 (378h58m34.704687196s) [noise=true , address=145.79.227.48:4303 , piece_id=44PNGUZODJGAD7AFPZGJE6XBDYCSN2CLPIARGZI62PQ7KKU7ANWQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.857890877 (378h58m34.728973912s) [noise=true , address=145.79.227.27:4691 , piece_id=PEBFRJ6FC4PNFEODQJFGBHHC56RKLUECXWW36YBHNO47HDCDXJWQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.991414468 (378h58m34.862497503s) [noise=true , address=45.91.116.7:4011 , piece_id=UH4YHD7JFAKXOXJTNKRAUITFZVFWU5AJG5XHSLCJ6JS7JCLWEVRQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:35.048985192 (378h58m34.920068237s) [noise=true , address=45.91.116.14:4189 , piece_id=NF2SZ7V2LDV76ADGQ6NXVNBHTHNLMRYS4SXW6BT4OM7BQ5TQ6BPQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:35.055578377 (378h58m34.926661412s) [noise=true , address=45.91.116.8:4048 , piece_id=3G4VLJ76W4UGBDWZEIQCBDFPRGAM7QL4GMC23LS73BC3JSFGGHPA, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:35.055594557 (378h58m34.926677602s) [seg_pos={0 0}, upload=332595]: Segment reached optimal threshold of 13 pieces. Dec 21 10:27:35.055773338 (378h58m34.926856373s) [noise=true , address=45.91.116.4:4940 , piece_id=L6QW3XENG6UBHKUUXTMOQQRIXBWBN6XXL7SHLY7W5KC5ES3SFS3A, seg_pos={0 0}, upload=332595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 21 10:27:35.055779740 (378h58m34.926862775s) [noise=true , address=45.91.116.4:4940 , piece_id=L6QW3XENG6UBHKUUXTMOQQRIXBWBN6XXL7SHLY7W5KC5ES3SFS3A, seg_pos={0 0}, upload=332595]: Long tail cancellation due to optimal threshold Dec 21 10:27:35.055834805 (378h58m34.92691785s) [noise=true , address=77.90.31.15:4700 , piece_id=WQP7T5QHGW23C7I5TBY7PESM5YTTXS3BAEH3RDRQTJZYHI37BUJQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 21 10:27:35.055845194 (378h58m34.926928239s) [noise=true , address=77.90.31.15:4700 , piece_id=WQP7T5QHGW23C7I5TBY7PESM5YTTXS3BAEH3RDRQTJZYHI37BUJQ, seg_pos={0 0}, upload=332595]: Long tail cancellation due to optimal threshold Dec 21 10:27:35.055982826 (378h58m34.927065862s) [noise=true , address=45.91.116.12:4225 , piece_id=WH4P44CPP53IJXXMGK5VK2E3PYBGFLJBWSCFSCBKR2QTALIIEGEA, seg_pos={0 0}, upload=332595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 21 10:27:35.056041539 (378h58m34.927124594s) [noise=true , address=45.91.116.12:4225 , piece_id=WH4P44CPP53IJXXMGK5VK2E3PYBGFLJBWSCFSCBKR2QTALIIEGEA, seg_pos={0 0}, upload=332595]: Long tail cancellation due to optimal threshold Dec 21 10:27:35.056064683 (378h58m34.927147728s) [seg_pos={0 0}, upload=332595]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 10:27:35.056146969 (378h58m34.927230004s) [upload=332595]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 10:27:35.056153121 (378h58m34.927236176s) [upload=332595]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 10:27:35.353226207 (378h58m35.224309242s) [upload=332595]: Done uploading segments... Dec 21 10:27:35.353243942 (378h58m35.224326977s) [upload=332595]: Upload finished. err: <nil> Duration: 5.892349468s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 5.64507183s trace: https://jaeger.dev.storj.io/trace/3ecb905698746e72 FAIL (2025-12-21 10:43:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: UH4YHD7JFAKXOXJTNKRAUITFZVFWU5AJG5XHSLCJ6JS7JCLWEVRQ) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 3G4VLJ76W4UGBDWZEIQCBDFPRGAM7QL4GMC23LS73BC3JSFGGHPA) context deadline exceeded ======LOGS====== Dec 21 10:27:32.520070050 (378h58m32.391153085s) [upload=332595]: Starting upload Dec 21 10:27:32.520134143 (378h58m32.391217178s) [upload=332595]: Done starting upload Dec 21 10:27:32.520194427 (378h58m32.391277472s) [upload=332595]: Uploading segments... Dec 21 10:27:32.520306451 (378h58m32.391389486s) [upload=332595]: Got next segment. Inline: false Dec 21 10:27:32.520335335 (378h58m32.39141839s) [upload=332595]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 10:27:32.520339694 (378h58m32.391422729s) [upload=332595]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 10:27:32.777396672 (378h58m32.648479718s) [seg_pos={0 0}, upload=332595]: Begin upload segment... Dec 21 10:27:32.777460795 (378h58m32.64854384s) [seg_pos={0 0}, upload=332595]: Done begin upload segment. Dec 21 10:27:32.777515899 (378h58m32.648598934s) [noise=true , address=145.79.227.23:4547 , piece_id=OOD2VMVRTCVGANLHQJME4HCJACBJEH2OUSKZBUY6RVNMSXA3CM7Q, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.777528564 (378h58m32.648611619s) [noise=true , address=145.79.227.27:4691 , piece_id=PEBFRJ6FC4PNFEODQJFGBHHC56RKLUECXWW36YBHNO47HDCDXJWQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.777605250 (378h58m32.648688295s) [noise=true , address=145.79.227.50:4373 , piece_id=GVHEJYGQ6A3HKPZ4V563E37K4L3CZUDEGCVJJHGNC722L3BNY46Q, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.778388482 (378h58m32.649471527s) [noise=true , address=145.79.227.39:4015 , piece_id=QB5S57M5HEYFEZOZYHBWMNFWVLQQEGHS73TBMANOJCCB42LK2AVA, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.778834370 (378h58m32.649917405s) [noise=true , address=45.91.116.12:4225 , piece_id=WH4P44CPP53IJXXMGK5VK2E3PYBGFLJBWSCFSCBKR2QTALIIEGEA, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.778858857 (378h58m32.649941892s) [noise=true , address=145.79.227.42:4121 , piece_id=FA7IZRDCG2XNN6PNDJOTZXGQXZWDOVV2XPR7MUSLK3YVYJVZE4UQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779235654 (378h58m32.650318689s) [noise=true , address=45.91.116.7:4011 , piece_id=UH4YHD7JFAKXOXJTNKRAUITFZVFWU5AJG5XHSLCJ6JS7JCLWEVRQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779269559 (378h58m32.650352594s) [noise=true , address=45.91.116.4:4940 , piece_id=L6QW3XENG6UBHKUUXTMOQQRIXBWBN6XXL7SHLY7W5KC5ES3SFS3A, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779536989 (378h58m32.650620024s) [noise=true , address=145.79.227.52:4445 , piece_id=YWZ6UHBVJWXLIGSE5HMV4WWZLLQLMBCB35WDSUFXK4XMBU5ZAT7Q, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779807954 (378h58m32.650890999s) [noise=true , address=77.90.31.15:4700 , piece_id=WQP7T5QHGW23C7I5TBY7PESM5YTTXS3BAEH3RDRQTJZYHI37BUJQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.780102345 (378h58m32.65118538s) [noise=true , address=145.79.227.17:4972 , piece_id=B3KYU5P3RL2BFR7QTS5ZK7B6UBDB3DTNTQKAGPM7B75S2SYEUFBQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.780482589 (378h58m32.651565634s) [noise=true , address=45.91.116.8:4048 , piece_id=3G4VLJ76W4UGBDWZEIQCBDFPRGAM7QL4GMC23LS73BC3JSFGGHPA, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.780863915 (378h58m32.65194696s) [noise=true , address=145.79.227.48:4303 , piece_id=44PNGUZODJGAD7AFPZGJE6XBDYCSN2CLPIARGZI62PQ7KKU7ANWQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.781783866 (378h58m32.652866911s) [noise=true , address=145.79.227.35:4482 , piece_id=3FUQ4OSF4B5DNWTAPY5W3LS2ZFBLCDWGXXLGASBKACSX3ONKLWWQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.782593317 (378h58m32.653676393s) [noise=true , address=45.91.116.14:4189 , piece_id=NF2SZ7V2LDV76ADGQ6NXVNBHTHNLMRYS4SXW6BT4OM7BQ5TQ6BPQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.783214190 (378h58m32.654297225s) [noise=true , address=145.79.227.51:4411 , piece_id=LC6PCUVNJN77OICI5JGXWN7LJD6JGMKKHHIQG6T4MKX5SU7FGRIQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:34.478418529 (378h58m34.349501564s) [upload=332595]: Next returned nil segment Dec 21 10:27:34.478430381 (378h58m34.349513416s) [upload=332595]: Waiting for error group managing segments... Dec 21 10:27:34.763620982 (378h58m34.634704027s) [noise=true , address=145.79.227.51:4411 , piece_id=LC6PCUVNJN77OICI5JGXWN7LJD6JGMKKHHIQG6T4MKX5SU7FGRIQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.770992478 (378h58m34.642075514s) [noise=true , address=145.79.227.50:4373 , piece_id=GVHEJYGQ6A3HKPZ4V563E37K4L3CZUDEGCVJJHGNC722L3BNY46Q, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.776530073 (378h58m34.647613108s) [noise=true , address=145.79.227.42:4121 , piece_id=FA7IZRDCG2XNN6PNDJOTZXGQXZWDOVV2XPR7MUSLK3YVYJVZE4UQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.786521177 (378h58m34.657604222s) [noise=true , address=145.79.227.17:4972 , piece_id=B3KYU5P3RL2BFR7QTS5ZK7B6UBDB3DTNTQKAGPM7B75S2SYEUFBQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.786591240 (378h58m34.657674275s) [noise=true , address=145.79.227.23:4547 , piece_id=OOD2VMVRTCVGANLHQJME4HCJACBJEH2OUSKZBUY6RVNMSXA3CM7Q, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.787618627 (378h58m34.658701672s) [noise=true , address=145.79.227.52:4445 , piece_id=YWZ6UHBVJWXLIGSE5HMV4WWZLLQLMBCB35WDSUFXK4XMBU5ZAT7Q, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.794650276 (378h58m34.665733311s) [noise=true , address=145.79.227.35:4482 , piece_id=3FUQ4OSF4B5DNWTAPY5W3LS2ZFBLCDWGXXLGASBKACSX3ONKLWWQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.794701083 (378h58m34.665784128s) [noise=true , address=145.79.227.39:4015 , piece_id=QB5S57M5HEYFEZOZYHBWMNFWVLQQEGHS73TBMANOJCCB42LK2AVA, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.833604161 (378h58m34.704687196s) [noise=true , address=145.79.227.48:4303 , piece_id=44PNGUZODJGAD7AFPZGJE6XBDYCSN2CLPIARGZI62PQ7KKU7ANWQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.857890877 (378h58m34.728973912s) [noise=true , address=145.79.227.27:4691 , piece_id=PEBFRJ6FC4PNFEODQJFGBHHC56RKLUECXWW36YBHNO47HDCDXJWQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.991414468 (378h58m34.862497503s) [noise=true , address=45.91.116.7:4011 , piece_id=UH4YHD7JFAKXOXJTNKRAUITFZVFWU5AJG5XHSLCJ6JS7JCLWEVRQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:35.048985192 (378h58m34.920068237s) [noise=true , address=45.91.116.14:4189 , piece_id=NF2SZ7V2LDV76ADGQ6NXVNBHTHNLMRYS4SXW6BT4OM7BQ5TQ6BPQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:35.055578377 (378h58m34.926661412s) [noise=true , address=45.91.116.8:4048 , piece_id=3G4VLJ76W4UGBDWZEIQCBDFPRGAM7QL4GMC23LS73BC3JSFGGHPA, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:35.055594557 (378h58m34.926677602s) [seg_pos={0 0}, upload=332595]: Segment reached optimal threshold of 13 pieces. Dec 21 10:27:35.055773338 (378h58m34.926856373s) [noise=true , address=45.91.116.4:4940 , piece_id=L6QW3XENG6UBHKUUXTMOQQRIXBWBN6XXL7SHLY7W5KC5ES3SFS3A, seg_pos={0 0}, upload=332595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 21 10:27:35.055779740 (378h58m34.926862775s) [noise=true , address=45.91.116.4:4940 , piece_id=L6QW3XENG6UBHKUUXTMOQQRIXBWBN6XXL7SHLY7W5KC5ES3SFS3A, seg_pos={0 0}, upload=332595]: Long tail cancellation due to optimal threshold Dec 21 10:27:35.055834805 (378h58m34.92691785s) [noise=true , address=77.90.31.15:4700 , piece_id=WQP7T5QHGW23C7I5TBY7PESM5YTTXS3BAEH3RDRQTJZYHI37BUJQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 21 10:27:35.055845194 (378h58m34.926928239s) [noise=true , address=77.90.31.15:4700 , piece_id=WQP7T5QHGW23C7I5TBY7PESM5YTTXS3BAEH3RDRQTJZYHI37BUJQ, seg_pos={0 0}, upload=332595]: Long tail cancellation due to optimal threshold Dec 21 10:27:35.055982826 (378h58m34.927065862s) [noise=true , address=45.91.116.12:4225 , piece_id=WH4P44CPP53IJXXMGK5VK2E3PYBGFLJBWSCFSCBKR2QTALIIEGEA, seg_pos={0 0}, upload=332595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 21 10:27:35.056041539 (378h58m34.927124594s) [noise=true , address=45.91.116.12:4225 , piece_id=WH4P44CPP53IJXXMGK5VK2E3PYBGFLJBWSCFSCBKR2QTALIIEGEA, seg_pos={0 0}, upload=332595]: Long tail cancellation due to optimal threshold Dec 21 10:27:35.056064683 (378h58m34.927147728s) [seg_pos={0 0}, upload=332595]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 10:27:35.056146969 (378h58m34.927230004s) [upload=332595]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 10:27:35.056153121 (378h58m34.927236176s) [upload=332595]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 10:27:35.353226207 (378h58m35.224309242s) [upload=332595]: Done uploading segments... Dec 21 10:27:35.353243942 (378h58m35.224326977s) [upload=332595]: Upload finished. err: <nil> Duration: 5.645091687s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 5.980105169s trace: https://jaeger.dev.storj.io/trace/3ecb905698746e72 FAIL (2025-12-21 10:42:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf, Piece ID: UH4YHD7JFAKXOXJTNKRAUITFZVFWU5AJG5XHSLCJ6JS7JCLWEVRQ) context deadline exceeded; piecestore: (Node ID: 1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB, Piece ID: 3G4VLJ76W4UGBDWZEIQCBDFPRGAM7QL4GMC23LS73BC3JSFGGHPA) context deadline exceeded ======LOGS====== Dec 21 10:27:32.520070050 (378h58m32.391153085s) [upload=332595]: Starting upload Dec 21 10:27:32.520134143 (378h58m32.391217178s) [upload=332595]: Done starting upload Dec 21 10:27:32.520194427 (378h58m32.391277472s) [upload=332595]: Uploading segments... Dec 21 10:27:32.520306451 (378h58m32.391389486s) [upload=332595]: Got next segment. Inline: false Dec 21 10:27:32.520335335 (378h58m32.39141839s) [upload=332595]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 21 10:27:32.520339694 (378h58m32.391422729s) [upload=332595]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 21 10:27:32.777396672 (378h58m32.648479718s) [seg_pos={0 0}, upload=332595]: Begin upload segment... Dec 21 10:27:32.777460795 (378h58m32.64854384s) [seg_pos={0 0}, upload=332595]: Done begin upload segment. Dec 21 10:27:32.777515899 (378h58m32.648598934s) [noise=true , address=145.79.227.23:4547 , piece_id=OOD2VMVRTCVGANLHQJME4HCJACBJEH2OUSKZBUY6RVNMSXA3CM7Q, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.777528564 (378h58m32.648611619s) [noise=true , address=145.79.227.27:4691 , piece_id=PEBFRJ6FC4PNFEODQJFGBHHC56RKLUECXWW36YBHNO47HDCDXJWQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.777605250 (378h58m32.648688295s) [noise=true , address=145.79.227.50:4373 , piece_id=GVHEJYGQ6A3HKPZ4V563E37K4L3CZUDEGCVJJHGNC722L3BNY46Q, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.778388482 (378h58m32.649471527s) [noise=true , address=145.79.227.39:4015 , piece_id=QB5S57M5HEYFEZOZYHBWMNFWVLQQEGHS73TBMANOJCCB42LK2AVA, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.778834370 (378h58m32.649917405s) [noise=true , address=45.91.116.12:4225 , piece_id=WH4P44CPP53IJXXMGK5VK2E3PYBGFLJBWSCFSCBKR2QTALIIEGEA, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.778858857 (378h58m32.649941892s) [noise=true , address=145.79.227.42:4121 , piece_id=FA7IZRDCG2XNN6PNDJOTZXGQXZWDOVV2XPR7MUSLK3YVYJVZE4UQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779235654 (378h58m32.650318689s) [noise=true , address=45.91.116.7:4011 , piece_id=UH4YHD7JFAKXOXJTNKRAUITFZVFWU5AJG5XHSLCJ6JS7JCLWEVRQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779269559 (378h58m32.650352594s) [noise=true , address=45.91.116.4:4940 , piece_id=L6QW3XENG6UBHKUUXTMOQQRIXBWBN6XXL7SHLY7W5KC5ES3SFS3A, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779536989 (378h58m32.650620024s) [noise=true , address=145.79.227.52:4445 , piece_id=YWZ6UHBVJWXLIGSE5HMV4WWZLLQLMBCB35WDSUFXK4XMBU5ZAT7Q, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.779807954 (378h58m32.650890999s) [noise=true , address=77.90.31.15:4700 , piece_id=WQP7T5QHGW23C7I5TBY7PESM5YTTXS3BAEH3RDRQTJZYHI37BUJQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.780102345 (378h58m32.65118538s) [noise=true , address=145.79.227.17:4972 , piece_id=B3KYU5P3RL2BFR7QTS5ZK7B6UBDB3DTNTQKAGPM7B75S2SYEUFBQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.780482589 (378h58m32.651565634s) [noise=true , address=45.91.116.8:4048 , piece_id=3G4VLJ76W4UGBDWZEIQCBDFPRGAM7QL4GMC23LS73BC3JSFGGHPA, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.780863915 (378h58m32.65194696s) [noise=true , address=145.79.227.48:4303 , piece_id=44PNGUZODJGAD7AFPZGJE6XBDYCSN2CLPIARGZI62PQ7KKU7ANWQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.781783866 (378h58m32.652866911s) [noise=true , address=145.79.227.35:4482 , piece_id=3FUQ4OSF4B5DNWTAPY5W3LS2ZFBLCDWGXXLGASBKACSX3ONKLWWQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.782593317 (378h58m32.653676393s) [noise=true , address=45.91.116.14:4189 , piece_id=NF2SZ7V2LDV76ADGQ6NXVNBHTHNLMRYS4SXW6BT4OM7BQ5TQ6BPQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:32.783214190 (378h58m32.654297225s) [noise=true , address=145.79.227.51:4411 , piece_id=LC6PCUVNJN77OICI5JGXWN7LJD6JGMKKHHIQG6T4MKX5SU7FGRIQ, seg_pos={0 0}, upload=332595]: Uploading piece... Dec 21 10:27:34.478418529 (378h58m34.349501564s) [upload=332595]: Next returned nil segment Dec 21 10:27:34.478430381 (378h58m34.349513416s) [upload=332595]: Waiting for error group managing segments... Dec 21 10:27:34.763620982 (378h58m34.634704027s) [noise=true , address=145.79.227.51:4411 , piece_id=LC6PCUVNJN77OICI5JGXWN7LJD6JGMKKHHIQG6T4MKX5SU7FGRIQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.770992478 (378h58m34.642075514s) [noise=true , address=145.79.227.50:4373 , piece_id=GVHEJYGQ6A3HKPZ4V563E37K4L3CZUDEGCVJJHGNC722L3BNY46Q, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.776530073 (378h58m34.647613108s) [noise=true , address=145.79.227.42:4121 , piece_id=FA7IZRDCG2XNN6PNDJOTZXGQXZWDOVV2XPR7MUSLK3YVYJVZE4UQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.786521177 (378h58m34.657604222s) [noise=true , address=145.79.227.17:4972 , piece_id=B3KYU5P3RL2BFR7QTS5ZK7B6UBDB3DTNTQKAGPM7B75S2SYEUFBQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.786591240 (378h58m34.657674275s) [noise=true , address=145.79.227.23:4547 , piece_id=OOD2VMVRTCVGANLHQJME4HCJACBJEH2OUSKZBUY6RVNMSXA3CM7Q, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.787618627 (378h58m34.658701672s) [noise=true , address=145.79.227.52:4445 , piece_id=YWZ6UHBVJWXLIGSE5HMV4WWZLLQLMBCB35WDSUFXK4XMBU5ZAT7Q, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.794650276 (378h58m34.665733311s) [noise=true , address=145.79.227.35:4482 , piece_id=3FUQ4OSF4B5DNWTAPY5W3LS2ZFBLCDWGXXLGASBKACSX3ONKLWWQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.794701083 (378h58m34.665784128s) [noise=true , address=145.79.227.39:4015 , piece_id=QB5S57M5HEYFEZOZYHBWMNFWVLQQEGHS73TBMANOJCCB42LK2AVA, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.833604161 (378h58m34.704687196s) [noise=true , address=145.79.227.48:4303 , piece_id=44PNGUZODJGAD7AFPZGJE6XBDYCSN2CLPIARGZI62PQ7KKU7ANWQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.857890877 (378h58m34.728973912s) [noise=true , address=145.79.227.27:4691 , piece_id=PEBFRJ6FC4PNFEODQJFGBHHC56RKLUECXWW36YBHNO47HDCDXJWQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:34.991414468 (378h58m34.862497503s) [noise=true , address=45.91.116.7:4011 , piece_id=UH4YHD7JFAKXOXJTNKRAUITFZVFWU5AJG5XHSLCJ6JS7JCLWEVRQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:35.048985192 (378h58m34.920068237s) [noise=true , address=45.91.116.14:4189 , piece_id=NF2SZ7V2LDV76ADGQ6NXVNBHTHNLMRYS4SXW6BT4OM7BQ5TQ6BPQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:35.055578377 (378h58m34.926661412s) [noise=true , address=45.91.116.8:4048 , piece_id=3G4VLJ76W4UGBDWZEIQCBDFPRGAM7QL4GMC23LS73BC3JSFGGHPA, seg_pos={0 0}, upload=332595]: Done uploading piece. err: <nil> Dec 21 10:27:35.055594557 (378h58m34.926677602s) [seg_pos={0 0}, upload=332595]: Segment reached optimal threshold of 13 pieces. Dec 21 10:27:35.055773338 (378h58m34.926856373s) [noise=true , address=45.91.116.4:4940 , piece_id=L6QW3XENG6UBHKUUXTMOQQRIXBWBN6XXL7SHLY7W5KC5ES3SFS3A, seg_pos={0 0}, upload=332595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ifieiAT4XHJHsEsrwSqqnRmctV79VUkoWwq3aCQjxWqK8MiCe): protocol: expected piece hash; context canceled; EOF Dec 21 10:27:35.055779740 (378h58m34.926862775s) [noise=true , address=45.91.116.4:4940 , piece_id=L6QW3XENG6UBHKUUXTMOQQRIXBWBN6XXL7SHLY7W5KC5ES3SFS3A, seg_pos={0 0}, upload=332595]: Long tail cancellation due to optimal threshold Dec 21 10:27:35.055834805 (378h58m34.92691785s) [noise=true , address=77.90.31.15:4700 , piece_id=WQP7T5QHGW23C7I5TBY7PESM5YTTXS3BAEH3RDRQTJZYHI37BUJQ, seg_pos={0 0}, upload=332595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q): protocol: expected piece hash; context canceled; EOF Dec 21 10:27:35.055845194 (378h58m34.926928239s) [noise=true , address=77.90.31.15:4700 , piece_id=WQP7T5QHGW23C7I5TBY7PESM5YTTXS3BAEH3RDRQTJZYHI37BUJQ, seg_pos={0 0}, upload=332595]: Long tail cancellation due to optimal threshold Dec 21 10:27:35.055982826 (378h58m34.927065862s) [noise=true , address=45.91.116.12:4225 , piece_id=WH4P44CPP53IJXXMGK5VK2E3PYBGFLJBWSCFSCBKR2QTALIIEGEA, seg_pos={0 0}, upload=332595]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: expected piece hash; context canceled; EOF Dec 21 10:27:35.056041539 (378h58m34.927124594s) [noise=true , address=45.91.116.12:4225 , piece_id=WH4P44CPP53IJXXMGK5VK2E3PYBGFLJBWSCFSCBKR2QTALIIEGEA, seg_pos={0 0}, upload=332595]: Long tail cancellation due to optimal threshold Dec 21 10:27:35.056064683 (378h58m34.927147728s) [seg_pos={0 0}, upload=332595]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 21 10:27:35.056146969 (378h58m34.927230004s) [upload=332595]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 21 10:27:35.056153121 (378h58m34.927236176s) [upload=332595]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 21 10:27:35.353226207 (378h58m35.224309242s) [upload=332595]: Done uploading segments... Dec 21 10:27:35.353243942 (378h58m35.224326977s) [upload=332595]: Upload finished. err: <nil> Duration: 5.980314806s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.AuthServiceHTTP{"https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777"} failure (antiblip window 3/60): (authservice.us_select_1)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)