Orbiter History

Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure

Orbiter Check History

Showing 50 of 900918 reports (Page 11326 of 18019)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [X.........X.....X......X.X.X.....X......XX.X..XXX.X.XXX.XX.X] duration: 30.004049878s trace: https://jaeger.dev.storj.io/trace/d35c52753bab26d FAIL (2025-12-28 15:29:47 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: YNM2MCCZ5USJZL5Q6JGU2RFKBPQZIIDOOONY7K3Z7AI3BWOX7APQ) context deadline exceeded; piecestore: (Node ID: 15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, Piece ID: 4GQUTLZQSAW4BG6PCPDWNUJ6RODQQLA7JAOEHXY7FM26AA4UW4OQ) context deadline exceeded; piecestore: (Node ID: 1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, Piece ID: VPHENVSNFXOS47AOHRV6IZGT2TIGMDI2HJC6ZHDQYNVYNNNNMRCQ) context deadline exceeded ======LOGS====== Dec 28 15:29:18.550107085 (552h0m18.42119013s) [upload=483966]: Starting upload Dec 28 15:29:18.550207204 (552h0m18.421290249s) [upload=483966]: Done starting upload Dec 28 15:29:18.550304450 (552h0m18.421387485s) [upload=483966]: Uploading segments... Dec 28 15:29:18.550520541 (552h0m18.421603576s) [upload=483966]: Got next segment. Inline: false Dec 28 15:29:18.550605603 (552h0m18.421688639s) [upload=483966]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 15:29:18.550613959 (552h0m18.421696994s) [upload=483966]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 15:29:18.793426946 (552h0m18.664509991s) [seg_pos={0 0}, upload=483966]: Begin upload segment... Dec 28 15:29:18.793504513 (552h0m18.664587569s) [seg_pos={0 0}, upload=483966]: Done begin upload segment. Dec 28 15:29:18.793560861 (552h0m18.664643906s) [noise=true , address=145.79.227.48:4301 , piece_id=QE4A2Z3V5S4GY6WGTSC2KIGJTUCVU2KJ4NE7SZUD5XBRHCPAL62Q, seg_pos={0 0}, upload=483966]: Uploading piece... Dec 28 15:29:18.793585568 (552h0m18.664668613s) [noise=true , address=77.90.31.10:4517 , piece_id=YNM2MCCZ5USJZL5Q6JGU2RFKBPQZIIDOOONY7K3Z7AI3BWOX7APQ, seg_pos={0 0}, upload=483966]: Uploading piece... Dec 28 15:29:18.794229163 (552h0m18.665312208s) [noise=true , address=145.79.227.2:4114 , piece_id=NEOFGARI2ZH6QJJT3D7Z6J6RNR5Z3QBGYJGCPD4PG2LRHKYRWZUQ, seg_pos={0 0}, upload=483966]: Uploading piece... Dec 28 15:29:18.794318794 (552h0m18.665401839s) [noise=true , address=145.79.227.11:4731 , piece_id=X4XVVPXLG3CSVEGSB7REXL2A4KTYYEUYUDONDXL4MBPKILT5H7HQ, seg_pos={0 0}, upload=483966]: Uploading piece... Dec 28 15:29:18.794728304 (552h0m18.665811339s) [noise=true , address=145.79.227.13:4802 , piece_id=I32RG5V26G2OMWULKJHIDJZZZMYK2IZXHWHYA7HLRPJSNHFUJQFQ, seg_pos={0 0}, upload=483966]: Uploading piece... Dec 28 15:29:18.795123777 (552h0m18.666206822s) [noise=true , address=145.79.227.15:4874 , piece_id=6FWGH55JCWFRYAKT2WDATAZWM7MJ7X3C2FYUFXRQHQAR53QVVE2A, seg_pos={0 0}, upload=483966]: Uploading piece... Dec 28 15:29:18.795581909 (552h0m18.666664944s) [noise=true , address=145.79.227.46:4601 , piece_id=GLV2IHEUWGKNTRP7BZJVP65DVNTKY3GVUJVILGJ3ELCESX7J6UUQ, seg_pos={0 0}, upload=483966]: Uploading piece... Dec 28 15:29:18.795920895 (552h0m18.66700394s) [noise=true , address=145.79.227.16:4910 , piece_id=4GQUTLZQSAW4BG6PCPDWNUJ6RODQQLA7JAOEHXY7FM26AA4UW4OQ, seg_pos={0 0}, upload=483966]: Uploading piece... Dec 28 15:29:18.796287052 (552h0m18.667370097s) [noise=true , address=145.79.227.37:4566 , piece_id=I4H7IYIZO3ZKSGN6TRB7F2EGUXK6GCDVM34YU7J4C6M2FRDYUJZA, seg_pos={0 0}, upload=483966]: Uploading piece... Dec 28 15:29:18.796301079 (552h0m18.667384124s) [noise=true , address=45.91.116.12:4227 , piece_id=E75QBVYYGED6W2E4GDDVYCZOP36YNFQHUUKBY65TS66EDA6EYSHA, seg_pos={0 0}, upload=483966]: Uploading piece... Dec 28 15:29:18.796754312 (552h0m18.667837357s) [noise=true , address=45.91.116.14:4191 , piece_id=VPHENVSNFXOS47AOHRV6IZGT2TIGMDI2HJC6ZHDQYNVYNNNNMRCQ, seg_pos={0 0}, upload=483966]: Uploading piece... Dec 28 15:29:18.797414068 (552h0m18.668497113s) [noise=true , address=145.79.227.51:4411 , piece_id=CCZ6JI7O7VOA4Y4N56D7QEOPG3B2JYL2LH6UKEHU36GPKU2CL3MA, seg_pos={0 0}, upload=483966]: Uploading piece... Dec 28 15:29:18.797832595 (552h0m18.66891564s) [noise=true , address=45.91.116.9:4261 , piece_id=PRWRPSFN4NG5XM3T253VK2GHKXLDZKKX57KTPFV44DKLJHVDWP5A, seg_pos={0 0}, upload=483966]: Uploading piece... Dec 28 15:29:18.798163355 (552h0m18.66924639s) [noise=true , address=145.79.227.14:4837 , piece_id=FCQSSBE6AXCXDHIV2TJ672LDAE5ULHT577443LMXEBU25HFARO2A, seg_pos={0 0}, upload=483966]: Uploading piece... Dec 28 15:29:18.798676863 (552h0m18.669759898s) [noise=true , address=145.79.227.45:4231 , piece_id=36NRYJQQNZT7MVJLRZKEUZQH7SAODK3XSJHGVHKN74GHP2JHBLMQ, seg_pos={0 0}, upload=483966]: Uploading piece... Dec 28 15:29:18.798930495 (552h0m18.67001353s) [noise=true , address=145.79.227.7:4294 , piece_id=PJPUBKOXSGNXIXCI5RCUJPDAVVTO5AZY6UGXJ6N57R6TPOPKGMOQ, seg_pos={0 0}, upload=483966]: Uploading piece... Dec 28 15:29:20.409434666 (552h0m20.280517701s) [upload=483966]: Next returned nil segment Dec 28 15:29:20.409448181 (552h0m20.280531216s) [upload=483966]: Waiting for error group managing segments... Dec 28 15:29:20.793709269 (552h0m20.664792304s) [noise=true , address=145.79.227.46:4601 , piece_id=GLV2IHEUWGKNTRP7BZJVP65DVNTKY3GVUJVILGJ3ELCESX7J6UUQ, seg_pos={0 0}, upload=483966]: Done uploading piece. err: <nil> Dec 28 15:29:20.793967570 (552h0m20.665050606s) [noise=true , address=145.79.227.11:4731 , piece_id=X4XVVPXLG3CSVEGSB7REXL2A4KTYYEUYUDONDXL4MBPKILT5H7HQ, seg_pos={0 0}, upload=483966]: Done uploading piece. err: <nil> Dec 28 15:29:20.794721897 (552h0m20.665804942s) [noise=true , address=145.79.227.15:4874 , piece_id=6FWGH55JCWFRYAKT2WDATAZWM7MJ7X3C2FYUFXRQHQAR53QVVE2A, seg_pos={0 0}, upload=483966]: Done uploading piece. err: <nil> Dec 28 15:29:20.795065451 (552h0m20.666148497s) [noise=true , address=145.79.227.37:4566 , piece_id=I4H7IYIZO3ZKSGN6TRB7F2EGUXK6GCDVM34YU7J4C6M2FRDYUJZA, seg_pos={0 0}, upload=483966]: Done uploading piece. err: <nil> Dec 28 15:29:20.799298231 (552h0m20.670381266s) [noise=true , address=145.79.227.16:4910 , piece_id=4GQUTLZQSAW4BG6PCPDWNUJ6RODQQLA7JAOEHXY7FM26AA4UW4OQ, seg_pos={0 0}, upload=483966]: Done uploading piece. err: <nil> Dec 28 15:29:20.799601298 (552h0m20.670684343s) [noise=true , address=145.79.227.48:4301 , piece_id=QE4A2Z3V5S4GY6WGTSC2KIGJTUCVU2KJ4NE7SZUD5XBRHCPAL62Q, seg_pos={0 0}, upload=483966]: Done uploading piece. err: <nil> Dec 28 15:29:20.800008364 (552h0m20.671091409s) [noise=true , address=145.79.227.13:4802 , piece_id=I32RG5V26G2OMWULKJHIDJZZZMYK2IZXHWHYA7HLRPJSNHFUJQFQ, seg_pos={0 0}, upload=483966]: Done uploading piece. err: <nil> Dec 28 15:29:20.800238531 (552h0m20.671321566s) [noise=true , address=145.79.227.2:4114 , piece_id=NEOFGARI2ZH6QJJT3D7Z6J6RNR5Z3QBGYJGCPD4PG2LRHKYRWZUQ, seg_pos={0 0}, upload=483966]: Done uploading piece. err: <nil> Dec 28 15:29:20.809204895 (552h0m20.68028793s) [noise=true , address=145.79.227.14:4837 , piece_id=FCQSSBE6AXCXDHIV2TJ672LDAE5ULHT577443LMXEBU25HFARO2A, seg_pos={0 0}, upload=483966]: Done uploading piece. err: <nil> Dec 28 15:29:20.814575201 (552h0m20.685658246s) [noise=true , address=145.79.227.7:4294 , piece_id=PJPUBKOXSGNXIXCI5RCUJPDAVVTO5AZY6UGXJ6N57R6TPOPKGMOQ, seg_pos={0 0}, upload=483966]: Done uploading piece. err: <nil> Dec 28 15:29:20.822258001 (552h0m20.693341036s) [noise=true , address=145.79.227.45:4231 , piece_id=36NRYJQQNZT7MVJLRZKEUZQH7SAODK3XSJHGVHKN74GHP2JHBLMQ, seg_pos={0 0}, upload=483966]: Done uploading piece. err: <nil> Dec 28 15:29:20.822302134 (552h0m20.693385179s) [noise=true , address=145.79.227.51:4411 , piece_id=CCZ6JI7O7VOA4Y4N56D7QEOPG3B2JYL2LH6UKEHU36GPKU2CL3MA, seg_pos={0 0}, upload=483966]: Done uploading piece. err: <nil> Dec 28 15:29:21.001266973 (552h0m20.872350018s) [noise=true , address=77.90.31.10:4517 , piece_id=YNM2MCCZ5USJZL5Q6JGU2RFKBPQZIIDOOONY7K3Z7AI3BWOX7APQ, seg_pos={0 0}, upload=483966]: Done uploading piece. err: <nil> Dec 28 15:29:21.001286491 (552h0m20.872369536s) [noise=true , address=45.91.116.14:4191 , piece_id=VPHENVSNFXOS47AOHRV6IZGT2TIGMDI2HJC6ZHDQYNVYNNNNMRCQ, seg_pos={0 0}, upload=483966]: Done uploading piece. err: <nil> Dec 28 15:29:21.001292672 (552h0m20.872375728s) [seg_pos={0 0}, upload=483966]: Segment reached optimal threshold of 13 pieces. Dec 28 15:29:21.001538801 (552h0m20.872621856s) [noise=true , address=45.91.116.9:4261 , piece_id=PRWRPSFN4NG5XM3T253VK2GHKXLDZKKX57KTPFV44DKLJHVDWP5A, seg_pos={0 0}, upload=483966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:59938->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:59938->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59938->45.91.116.9:4261: use of closed network connection Dec 28 15:29:21.001548800 (552h0m20.872631855s) [noise=true , address=45.91.116.9:4261 , piece_id=PRWRPSFN4NG5XM3T253VK2GHKXLDZKKX57KTPFV44DKLJHVDWP5A, seg_pos={0 0}, upload=483966]: Long tail cancellation due to optimal threshold Dec 28 15:29:21.001762877 (552h0m20.872845962s) [noise=true , address=45.91.116.12:4227 , piece_id=E75QBVYYGED6W2E4GDDVYCZOP36YNFQHUUKBY65TS66EDA6EYSHA, seg_pos={0 0}, upload=483966]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: expected piece hash; context canceled; EOF Dec 28 15:29:21.001773168 (552h0m20.872856223s) [noise=true , address=45.91.116.12:4227 , piece_id=E75QBVYYGED6W2E4GDDVYCZOP36YNFQHUUKBY65TS66EDA6EYSHA, seg_pos={0 0}, upload=483966]: Long tail cancellation due to optimal threshold Dec 28 15:29:21.001823172 (552h0m20.872906237s) [seg_pos={0 0}, upload=483966]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Dec 28 15:29:21.001922922 (552h0m20.873005988s) [upload=483966]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 15:29:21.001929555 (552h0m20.8730126s) [upload=483966]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 15:29:21.304959155 (552h0m21.17604222s) [upload=483966]: Done uploading segments... Dec 28 15:29:21.304983640 (552h0m21.176066686s) [upload=483966]: Upload finished. err: <nil> Duration: 30.004075495s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 13.179561052s trace: https://jaeger.dev.storj.io/trace/4ce8174d0eaf2b7b FAIL (2025-12-28 15:29:29 UTC) orbiter warning: context deadline exceeded Duration: 13.17958639s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 42 out of 60 is failed (threshold=6) [XXXXX.X.X.XXXXX...X.X.XX.XXX.X.XXXXX.XX.XX.X.XXXXX.X.XXX.XXX] duration: 5.000929919s trace: https://jaeger.dev.storj.io/trace/383e43ee18f9fc0b FAIL (2025-12-28 15:29:27 UTC) orbiter warning: context deadline exceeded Duration: 5.000950407s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 5 out of 60 is failed (threshold=3) [......X....X..........................X....X...X............] duration: 50.59487605s trace: https://jaeger.dev.storj.io/trace/51e2216a4efa8b95 FAIL (2025-12-28 15:29:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 15s) [noise=true , address=145.79.227.40:4083 , piece_id=RSMZDEGC4MX5TFRFKI33WNDK42NZDQ4M6UPRQWO4GE6VWGCJ6AFA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.854006008 (551h48m43.725089043s) [noise=true , address=77.90.31.12:4610 , piece_id=RSI6EFWFTX4JZUBEFTV3VPMJWMWI7VDMUHG6DMK7CSKTH7MDFGXA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.854333892 (551h48m43.725416937s) [noise=true , address=64.49.13.139:4476 , piece_id=FUSCXA3KPS24G3R5ORJTDOUOU2KMIK5SNBPROHL2CRKMKBRWFN6A, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.854753131 (551h48m43.725836176s) [noise=true , address=145.79.227.34:4932 , piece_id=5NGLBPM6NRZIR4NJTA43KIFQQTD6DS6B37DIFCHXGT5YMDGNMXKA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.855102245 (551h48m43.726185381s) [noise=true , address=77.90.31.15:4727 , piece_id=YJDXXQTMZQRYCEAKLZUFPW6JO7RWHJMJAOFMURYFNLQSJ4LJG2CQ, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.855467040 (551h48m43.726550076s) [noise=true , address=145.79.227.6:4280 , piece_id=ARMYEAFMP6DPW2JSHWHOYPJ6OUUSUQETJCSSQQXFQX4DDCVZYLYQ, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.855834780 (551h48m43.726917815s) [noise=true , address=193.149.164.143:4694 , piece_id=SDK5DIHWVFOZQIHQWJMZLOTQJEKRDBUTQ7R7ZTKOKF5LGECJRMOA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.856263167 (551h48m43.727346212s) [noise=true , address=145.79.227.43:4167 , piece_id=WEXP7LDX6O7KPXA6XIMRWNTMGNGSTE3XRFODPNMWFKR3D5WXX3LQ, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.858002919 (551h48m43.729085964s) [noise=true , address=145.79.227.30:4766 , piece_id=YA4HOOMXQRQCPE3VMQ6WEXBM3ZIED4RZLVFHZHRLBQJFQBFP2F7A, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.858063876 (551h48m43.729146911s) [noise=true , address=64.49.13.169:4716 , piece_id=V3PN2MUBRFNCLCYOVHYLXUORQO4S2TLRL6HMGJQCMJ6PCBZISPKQ, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.859819067 (551h48m43.730902113s) [noise=true , address=145.79.227.49:4367 , piece_id=CITJCIKXK4LYSUV6QRTLIHB7SEH5QOPUM7XR4JDHV54EJ5FLLMDQ, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.861538431 (551h48m43.732621466s) [noise=true , address=145.79.227.33:4904 , piece_id=4B64RHS2HQ6WZYND6MZGU4AZC4LQBXPNKOE2ABNVZQTEBXALQCDA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.863179095 (551h48m43.73426214s) [noise=true , address=77.90.31.19:4874 , piece_id=IXOOASSSDYJS42HWYZKK4PKE4LW47BT4WF5WRRXY7ANPLY632LRA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.864788100 (551h48m43.735871145s) [noise=true , address=77.90.31.16:4749 , piece_id=IVHKFK76TGLMJ7IBVS5ZCCMB35LKHF4FE5HLE7P5QAVGAWGSRHNA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.866240675 (551h48m43.73732371s) [noise=true , address=45.91.116.7:4029 , piece_id=VTKGXA2SWJZHOLLWREHQO66OHQZWHVRK25STFR62PEIJFGD4SG4A, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.867523768 (551h48m43.738606803s) [noise=true , address=45.91.116.6:4998 , piece_id=KCEYZDX2WGV7UCYX4H4BNSR2CWOHGPSEW76LKU2BGC3UDHA5M5SA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:45.143085883 (551h48m45.014168928s) [upload=483809]: Next returned nil segment Dec 28 15:17:45.143099779 (551h48m45.014182824s) [upload=483809]: Waiting for error group managing segments... Dec 28 15:17:45.472957433 (551h48m45.344040468s) [noise=true , address=64.49.13.139:4476 , piece_id=FUSCXA3KPS24G3R5ORJTDOUOU2KMIK5SNBPROHL2CRKMKBRWFN6A, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:45.739300305 (551h48m45.61038334s) [noise=true , address=145.79.227.6:4280 , piece_id=ARMYEAFMP6DPW2JSHWHOYPJ6OUUSUQETJCSSQQXFQX4DDCVZYLYQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:45.743884264 (551h48m45.614967309s) [noise=true , address=145.79.227.21:4508 , piece_id=PGAOV6UBGJPU7M5L3IR2LQU5LSISUJVKKBB7ZRV233F5MBEAWK6A, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:45.936004292 (551h48m45.807087328s) [noise=true , address=77.90.31.15:4727 , piece_id=YJDXXQTMZQRYCEAKLZUFPW6JO7RWHJMJAOFMURYFNLQSJ4LJG2CQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:45.968797306 (551h48m45.839880351s) [noise=true , address=77.90.31.16:4749 , piece_id=IVHKFK76TGLMJ7IBVS5ZCCMB35LKHF4FE5HLE7P5QAVGAWGSRHNA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.007515762 (551h48m45.878598797s) [noise=true , address=77.90.31.11:4580 , piece_id=UUPDJ75SL3HOJREC2SXJGQMUVLLP5WLN2L3LEON6KIKPB24DEKPQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.147397009 (551h48m46.018480054s) [noise=true , address=45.91.116.7:4029 , piece_id=VTKGXA2SWJZHOLLWREHQO66OHQZWHVRK25STFR62PEIJFGD4SG4A, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.460756007 (551h48m46.331839042s) [noise=true , address=193.149.164.143:4694 , piece_id=SDK5DIHWVFOZQIHQWJMZLOTQJEKRDBUTQ7R7ZTKOKF5LGECJRMOA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.662946081 (551h48m46.534029127s) [noise=true , address=145.79.227.43:4167 , piece_id=WEXP7LDX6O7KPXA6XIMRWNTMGNGSTE3XRFODPNMWFKR3D5WXX3LQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.695276604 (551h48m46.566359639s) [noise=true , address=45.91.116.6:4998 , piece_id=KCEYZDX2WGV7UCYX4H4BNSR2CWOHGPSEW76LKU2BGC3UDHA5M5SA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.739405984 (551h48m46.610489029s) [noise=true , address=145.79.227.33:4904 , piece_id=4B64RHS2HQ6WZYND6MZGU4AZC4LQBXPNKOE2ABNVZQTEBXALQCDA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.764293434 (551h48m46.635376479s) [noise=true , address=145.79.227.40:4083 , piece_id=RSMZDEGC4MX5TFRFKI33WNDK42NZDQ4M6UPRQWO4GE6VWGCJ6AFA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.771299524 (551h48m46.64238256s) [noise=true , address=145.79.227.30:4766 , piece_id=YA4HOOMXQRQCPE3VMQ6WEXBM3ZIED4RZLVFHZHRLBQJFQBFP2F7A, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.791263379 (551h48m46.662346414s) [noise=true , address=77.90.31.19:4874 , piece_id=IXOOASSSDYJS42HWYZKK4PKE4LW47BT4WF5WRRXY7ANPLY632LRA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.791282065 (551h48m46.6623651s) [seg_pos={0 0}, upload=483809]: Segment reached optimal threshold of 14 pieces. Dec 28 15:17:46.791451286 (551h48m46.662534321s) [noise=true , address=145.79.227.49:4367 , piece_id=CITJCIKXK4LYSUV6QRTLIHB7SEH5QOPUM7XR4JDHV54EJ5FLLMDQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF Dec 28 15:17:46.791458450 (551h48m46.662541496s) [noise=true , address=145.79.227.49:4367 , piece_id=CITJCIKXK4LYSUV6QRTLIHB7SEH5QOPUM7XR4JDHV54EJ5FLLMDQ, seg_pos={0 0}, upload=483809]: Long tail cancellation due to optimal threshold Dec 28 15:17:46.791491023 (551h48m46.662574058s) [noise=true , address=77.90.31.12:4610 , piece_id=RSI6EFWFTX4JZUBEFTV3VPMJWMWI7VDMUHG6DMK7CSKTH7MDFGXA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi): protocol: expected piece hash; context canceled; EOF Dec 28 15:17:46.791502935 (551h48m46.66258598s) [noise=true , address=77.90.31.12:4610 , piece_id=RSI6EFWFTX4JZUBEFTV3VPMJWMWI7VDMUHG6DMK7CSKTH7MDFGXA, seg_pos={0 0}, upload=483809]: Long tail cancellation due to optimal threshold Dec 28 15:17:46.791556076 (551h48m46.662639111s) [noise=true , address=145.79.227.34:4932 , piece_id=5NGLBPM6NRZIR4NJTA43KIFQQTD6DS6B37DIFCHXGT5YMDGNMXKA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qJfPZErJx4XGBw7NmfCLS7BogRnN29rUibuWkzQDFUNvPBpmm): protocol: expected piece hash; context canceled; EOF Dec 28 15:17:46.791561086 (551h48m46.662644131s) [noise=true , address=145.79.227.34:4932 , piece_id=5NGLBPM6NRZIR4NJTA43KIFQQTD6DS6B37DIFCHXGT5YMDGNMXKA, seg_pos={0 0}, upload=483809]: Long tail cancellation due to optimal threshold Dec 28 15:17:46.791659873 (551h48m46.662742908s) [noise=true , address=64.49.13.169:4716 , piece_id=V3PN2MUBRFNCLCYOVHYLXUORQO4S2TLRL6HMGJQCMJ6PCBZISPKQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:48986->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:48986->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48986->64.49.13.169:4716: use of closed network connection Dec 28 15:17:46.791665625 (551h48m46.66274866s) [noise=true , address=64.49.13.169:4716 , piece_id=V3PN2MUBRFNCLCYOVHYLXUORQO4S2TLRL6HMGJQCMJ6PCBZISPKQ, seg_pos={0 0}, upload=483809]: Long tail cancellation due to optimal threshold Dec 28 15:17:46.791739836 (551h48m46.662822881s) [seg_pos={0 0}, upload=483809]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 15:17:46.791808337 (551h48m46.662891372s) [upload=483809]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 15:17:46.791812865 (551h48m46.6628959s) [upload=483809]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 15:17:47.097154206 (551h48m46.968237251s) [upload=483809]: Done uploading segments... Dec 28 15:17:47.097173253 (551h48m46.968256288s) [upload=483809]: Upload finished. err: <nil> Duration: 50.594902239s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 41 out of 60 is failed (threshold=6) [.XXXXX.X.X.XXXXX...X.X.XX.XXX.X.XXXXX.XX.XX.X.XXXXX.X.XXX.XX] duration: 5.001877331s trace: https://jaeger.dev.storj.io/trace/4c7b36c8f495ec4f FAIL (2025-12-28 15:28:56 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001902247s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.X.........X.....X......X.X.X.....X......XX.X..XXX.X.XXX.XX.] duration: 5.959214099s trace: https://jaeger.dev.storj.io/trace/56a60bfd907dd0ce FAIL (2025-12-28 15:28:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: XKWIUHSAUTEYYPNNRI6ZUE6Q2A7IU6FU4E5O26ZZY744I4URTVKQ) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: GIZKJMGYLBZ6V4EXP6G5SUVKJ5RU6HSOXIRG2PNGLTUXBNQNY3SA) context deadline exceeded ======LOGS====== Dec 28 15:27:20.752243379 (551h58m20.623326424s) [upload=483942]: Starting upload Dec 28 15:27:20.752314755 (551h58m20.62339781s) [upload=483942]: Done starting upload Dec 28 15:27:20.752395438 (551h58m20.623478473s) [upload=483942]: Uploading segments... Dec 28 15:27:20.752518202 (551h58m20.623601237s) [upload=483942]: Got next segment. Inline: false Dec 28 15:27:20.752554341 (551h58m20.623637386s) [upload=483942]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 15:27:20.752560202 (551h58m20.623643237s) [upload=483942]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 15:27:21.015134759 (551h58m20.886217804s) [seg_pos={0 0}, upload=483942]: Begin upload segment... Dec 28 15:27:21.015190095 (551h58m20.88627313s) [seg_pos={0 0}, upload=483942]: Done begin upload segment. Dec 28 15:27:21.015208369 (551h58m20.886291414s) [noise=true , address=145.79.227.38:4978 , piece_id=3G5ELCFYP626AAYGWBFGK7J5LMUUYJH3JAEY2BK3RIIUP2DFAWNA, seg_pos={0 0}, upload=483942]: Uploading piece... Dec 28 15:27:21.015223990 (551h58m20.886307025s) [noise=true , address=145.79.227.42:4121 , piece_id=YDYGBBEIFYTRDOVVD2AOGHVSK4CGGTXJD4EGGRSVHOLN7UGFO3NA, seg_pos={0 0}, upload=483942]: Uploading piece... Dec 28 15:27:21.016151024 (551h58m20.887234059s) [noise=true , address=77.90.31.21:4878 , piece_id=HO3LD7QLHWBKPOVGKHGTYZUKYQBL7VBOZKZDJZZW4FQ4N5EUMIEQ, seg_pos={0 0}, upload=483942]: Uploading piece... Dec 28 15:27:21.016595320 (551h58m20.887678365s) [noise=true , address=145.79.227.31:4798 , piece_id=NGUUZGXFKFAHUJLRQBPEY7B6VM5N4EDODLSXQWOYFNQKI6AYBOFQ, seg_pos={0 0}, upload=483942]: Uploading piece... Dec 28 15:27:21.016973921 (551h58m20.888056957s) [noise=true , address=145.79.227.50:4373 , piece_id=CLYDBUY6NO4E2KQG6PJNX6OIWYMQQ5GXF74KZGFGLDO4LC6DEKOQ, seg_pos={0 0}, upload=483942]: Uploading piece... Dec 28 15:27:21.017373893 (551h58m20.888456928s) [noise=true , address=77.90.31.14:4662 , piece_id=XKWIUHSAUTEYYPNNRI6ZUE6Q2A7IU6FU4E5O26ZZY744I4URTVKQ, seg_pos={0 0}, upload=483942]: Uploading piece... Dec 28 15:27:21.017786248 (551h58m20.888869283s) [noise=true , address=145.79.227.7:4294 , piece_id=KMGMXRNCANSOALFRLCIOKG2R4DCOSOGBIAGMFFSVR66BI3P5T7LQ, seg_pos={0 0}, upload=483942]: Uploading piece... Dec 28 15:27:21.018224803 (551h58m20.889307838s) [noise=true , address=145.79.227.4:4186 , piece_id=LY4WKJWSS3DMFCVYMXR5Q3GBFPGVFSHICCMEY4RNBGREHAFTGMLA, seg_pos={0 0}, upload=483942]: Uploading piece... Dec 28 15:27:21.019557892 (551h58m20.890640927s) [noise=true , address=77.90.31.18:4808 , piece_id=KUGXOQZZJFGD7TJCDVJ7ZBVSHWXHCHUPDZXN4RJASJLJYCKEV6AQ, seg_pos={0 0}, upload=483942]: Uploading piece... Dec 28 15:27:21.019767200 (551h58m20.890850235s) [noise=true , address=45.91.116.8:4046 , piece_id=GIZKJMGYLBZ6V4EXP6G5SUVKJ5RU6HSOXIRG2PNGLTUXBNQNY3SA, seg_pos={0 0}, upload=483942]: Uploading piece... Dec 28 15:27:21.020905919 (551h58m20.891988954s) [noise=true , address=45.91.116.14:4192 , piece_id=IDFMTTJGEPVIAQDONMWNBYDTURMPMZRQHJTHM3BJPPHBHF6GA42Q, seg_pos={0 0}, upload=483942]: Uploading piece... Dec 28 15:27:21.021227922 (551h58m20.892310967s) [noise=true , address=145.79.227.6:4258 , piece_id=GGQFGT44NK2L5T4H3Z3AWUNAXOZWUZXQJLL5NYCITNQNZ3DHOJSA, seg_pos={0 0}, upload=483942]: Uploading piece... Dec 28 15:27:21.022328337 (551h58m20.893411382s) [noise=true , address=145.79.227.14:4840 , piece_id=7JB5AK7OQBRGBLAA6YG6NMEEJJGQBH6EF3FFGZZFNVLCQU7ZP4UQ, seg_pos={0 0}, upload=483942]: Uploading piece... Dec 28 15:27:21.022693492 (551h58m20.893776527s) [noise=true , address=145.79.227.23:4546 , piece_id=UXUPM3THNJWF6WPZIB4PUQOLCXHJMHYMHS7DTQW4AYQZMVJKP2QQ, seg_pos={0 0}, upload=483942]: Uploading piece... Dec 28 15:27:21.023687896 (551h58m20.894770931s) [noise=true , address=145.79.227.17:4972 , piece_id=DFSNJUAMWENCCYTT4DNBLFB5I5SQG74HL5WSHC5TZ5IHAEYCBEZQ, seg_pos={0 0}, upload=483942]: Uploading piece... Dec 28 15:27:21.024042040 (551h58m20.895125075s) [noise=true , address=145.79.227.11:4730 , piece_id=OL23QJIWRUM6GDVANS5PZSXVF4WBFYYR7T5X7YQH2ZFWGOD7D5GA, seg_pos={0 0}, upload=483942]: Uploading piece... Dec 28 15:27:22.735609381 (551h58m22.606692416s) [upload=483942]: Next returned nil segment Dec 28 15:27:22.735628217 (551h58m22.606711252s) [upload=483942]: Waiting for error group managing segments... Dec 28 15:27:23.029257516 (551h58m22.900340551s) [noise=true , address=145.79.227.42:4121 , piece_id=YDYGBBEIFYTRDOVVD2AOGHVSK4CGGTXJD4EGGRSVHOLN7UGFO3NA, seg_pos={0 0}, upload=483942]: Done uploading piece. err: <nil> Dec 28 15:27:23.030924962 (551h58m22.902007997s) [noise=true , address=145.79.227.50:4373 , piece_id=CLYDBUY6NO4E2KQG6PJNX6OIWYMQQ5GXF74KZGFGLDO4LC6DEKOQ, seg_pos={0 0}, upload=483942]: Done uploading piece. err: <nil> Dec 28 15:27:23.033555650 (551h58m22.904638685s) [noise=true , address=145.79.227.23:4546 , piece_id=UXUPM3THNJWF6WPZIB4PUQOLCXHJMHYMHS7DTQW4AYQZMVJKP2QQ, seg_pos={0 0}, upload=483942]: Done uploading piece. err: <nil> Dec 28 15:27:23.036308322 (551h58m22.907391357s) [noise=true , address=145.79.227.31:4798 , piece_id=NGUUZGXFKFAHUJLRQBPEY7B6VM5N4EDODLSXQWOYFNQKI6AYBOFQ, seg_pos={0 0}, upload=483942]: Done uploading piece. err: <nil> Dec 28 15:27:23.036565863 (551h58m22.907648898s) [noise=true , address=145.79.227.38:4978 , piece_id=3G5ELCFYP626AAYGWBFGK7J5LMUUYJH3JAEY2BK3RIIUP2DFAWNA, seg_pos={0 0}, upload=483942]: Done uploading piece. err: <nil> Dec 28 15:27:23.040755891 (551h58m22.911838926s) [noise=true , address=145.79.227.17:4972 , piece_id=DFSNJUAMWENCCYTT4DNBLFB5I5SQG74HL5WSHC5TZ5IHAEYCBEZQ, seg_pos={0 0}, upload=483942]: Done uploading piece. err: <nil> Dec 28 15:27:23.044459984 (551h58m22.915543019s) [noise=true , address=145.79.227.6:4258 , piece_id=GGQFGT44NK2L5T4H3Z3AWUNAXOZWUZXQJLL5NYCITNQNZ3DHOJSA, seg_pos={0 0}, upload=483942]: Done uploading piece. err: <nil> Dec 28 15:27:23.044704419 (551h58m22.915787454s) [noise=true , address=145.79.227.14:4840 , piece_id=7JB5AK7OQBRGBLAA6YG6NMEEJJGQBH6EF3FFGZZFNVLCQU7ZP4UQ, seg_pos={0 0}, upload=483942]: Done uploading piece. err: <nil> Dec 28 15:27:23.050297249 (551h58m22.921380284s) [noise=true , address=145.79.227.7:4294 , piece_id=KMGMXRNCANSOALFRLCIOKG2R4DCOSOGBIAGMFFSVR66BI3P5T7LQ, seg_pos={0 0}, upload=483942]: Done uploading piece. err: <nil> Dec 28 15:27:23.052311224 (551h58m22.923394259s) [noise=true , address=145.79.227.4:4186 , piece_id=LY4WKJWSS3DMFCVYMXR5Q3GBFPGVFSHICCMEY4RNBGREHAFTGMLA, seg_pos={0 0}, upload=483942]: Done uploading piece. err: <nil> Dec 28 15:27:23.264045862 (551h58m23.135128897s) [noise=true , address=77.90.31.14:4662 , piece_id=XKWIUHSAUTEYYPNNRI6ZUE6Q2A7IU6FU4E5O26ZZY744I4URTVKQ, seg_pos={0 0}, upload=483942]: Done uploading piece. err: <nil> Dec 28 15:27:23.302920026 (551h58m23.174003061s) [noise=true , address=45.91.116.14:4192 , piece_id=IDFMTTJGEPVIAQDONMWNBYDTURMPMZRQHJTHM3BJPPHBHF6GA42Q, seg_pos={0 0}, upload=483942]: Done uploading piece. err: <nil> Dec 28 15:27:23.310694249 (551h58m23.181777284s) [noise=true , address=45.91.116.8:4046 , piece_id=GIZKJMGYLBZ6V4EXP6G5SUVKJ5RU6HSOXIRG2PNGLTUXBNQNY3SA, seg_pos={0 0}, upload=483942]: Done uploading piece. err: <nil> Dec 28 15:27:23.310715169 (551h58m23.181798204s) [seg_pos={0 0}, upload=483942]: Segment reached optimal threshold of 13 pieces. Dec 28 15:27:23.311021762 (551h58m23.182104807s) [noise=true , address=77.90.31.21:4878 , piece_id=HO3LD7QLHWBKPOVGKHGTYZUKYQBL7VBOZKZDJZZW4FQ4N5EUMIEQ, seg_pos={0 0}, upload=483942]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 28 15:27:23.311040618 (551h58m23.182123653s) [noise=true , address=77.90.31.21:4878 , piece_id=HO3LD7QLHWBKPOVGKHGTYZUKYQBL7VBOZKZDJZZW4FQ4N5EUMIEQ, seg_pos={0 0}, upload=483942]: Long tail cancellation due to optimal threshold Dec 28 15:27:23.311065306 (551h58m23.182148351s) [noise=true , address=77.90.31.18:4808 , piece_id=KUGXOQZZJFGD7TJCDVJ7ZBVSHWXHCHUPDZXN4RJASJLJYCKEV6AQ, seg_pos={0 0}, upload=483942]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: read tcp 188.166.225.170:40802->77.90.31.18:4808: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 15:27:23.311072559 (551h58m23.182155594s) [noise=true , address=77.90.31.18:4808 , piece_id=KUGXOQZZJFGD7TJCDVJ7ZBVSHWXHCHUPDZXN4RJASJLJYCKEV6AQ, seg_pos={0 0}, upload=483942]: Long tail cancellation due to optimal threshold Dec 28 15:27:23.311249737 (551h58m23.182332772s) [noise=true , address=145.79.227.11:4730 , piece_id=OL23QJIWRUM6GDVANS5PZSXVF4WBFYYR7T5X7YQH2ZFWGOD7D5GA, seg_pos={0 0}, upload=483942]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom): protocol: expected piece hash; context canceled; EOF Dec 28 15:27:23.311258303 (551h58m23.182341338s) [noise=true , address=145.79.227.11:4730 , piece_id=OL23QJIWRUM6GDVANS5PZSXVF4WBFYYR7T5X7YQH2ZFWGOD7D5GA, seg_pos={0 0}, upload=483942]: Long tail cancellation due to optimal threshold Dec 28 15:27:23.311275536 (551h58m23.182358581s) [seg_pos={0 0}, upload=483942]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 15:27:23.311347533 (551h58m23.182430568s) [upload=483942]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 15:27:23.311352362 (551h58m23.182435407s) [upload=483942]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 15:27:23.600575321 (551h58m23.471658366s) [upload=483942]: Done uploading segments... Dec 28 15:27:23.600594657 (551h58m23.471677702s) [upload=483942]: Upload finished. err: <nil> Duration: 5.959241743s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 40 out of 60 is failed (threshold=6) [..XXXXX.X.X.XXXXX...X.X.XX.XXX.X.XXXXX.XX.XX.X.XXXXX.X.XXX.X] duration: 5.001935402s trace: https://jaeger.dev.storj.io/trace/65e0dd1e403eea09 FAIL (2025-12-28 15:28:38 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001962344s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 18.633257682s trace: https://jaeger.dev.storj.io/trace/4ce8174d0eaf2b7b FAIL (2025-12-28 15:28:21 UTC) orbiter warning: context deadline exceeded Duration: 18.633285886s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 40 out of 60 is failed (threshold=6) [X..XXXXX.X.X.XXXXX...X.X.XX.XXX.X.XXXXX.XX.XX.X.XXXXX.X.XXX.] duration: 5.00101517s trace: https://jaeger.dev.storj.io/trace/122d3f5514e3118f FAIL (2025-12-28 15:27:59 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001038454s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [X......X....X..........................X....X...X...........] duration: 40.409598652s trace: https://jaeger.dev.storj.io/trace/51e2216a4efa8b95 FAIL (2025-12-28 15:27:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 15s) [noise=true , address=145.79.227.40:4083 , piece_id=RSMZDEGC4MX5TFRFKI33WNDK42NZDQ4M6UPRQWO4GE6VWGCJ6AFA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.854006008 (551h48m43.725089043s) [noise=true , address=77.90.31.12:4610 , piece_id=RSI6EFWFTX4JZUBEFTV3VPMJWMWI7VDMUHG6DMK7CSKTH7MDFGXA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.854333892 (551h48m43.725416937s) [noise=true , address=64.49.13.139:4476 , piece_id=FUSCXA3KPS24G3R5ORJTDOUOU2KMIK5SNBPROHL2CRKMKBRWFN6A, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.854753131 (551h48m43.725836176s) [noise=true , address=145.79.227.34:4932 , piece_id=5NGLBPM6NRZIR4NJTA43KIFQQTD6DS6B37DIFCHXGT5YMDGNMXKA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.855102245 (551h48m43.726185381s) [noise=true , address=77.90.31.15:4727 , piece_id=YJDXXQTMZQRYCEAKLZUFPW6JO7RWHJMJAOFMURYFNLQSJ4LJG2CQ, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.855467040 (551h48m43.726550076s) [noise=true , address=145.79.227.6:4280 , piece_id=ARMYEAFMP6DPW2JSHWHOYPJ6OUUSUQETJCSSQQXFQX4DDCVZYLYQ, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.855834780 (551h48m43.726917815s) [noise=true , address=193.149.164.143:4694 , piece_id=SDK5DIHWVFOZQIHQWJMZLOTQJEKRDBUTQ7R7ZTKOKF5LGECJRMOA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.856263167 (551h48m43.727346212s) [noise=true , address=145.79.227.43:4167 , piece_id=WEXP7LDX6O7KPXA6XIMRWNTMGNGSTE3XRFODPNMWFKR3D5WXX3LQ, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.858002919 (551h48m43.729085964s) [noise=true , address=145.79.227.30:4766 , piece_id=YA4HOOMXQRQCPE3VMQ6WEXBM3ZIED4RZLVFHZHRLBQJFQBFP2F7A, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.858063876 (551h48m43.729146911s) [noise=true , address=64.49.13.169:4716 , piece_id=V3PN2MUBRFNCLCYOVHYLXUORQO4S2TLRL6HMGJQCMJ6PCBZISPKQ, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.859819067 (551h48m43.730902113s) [noise=true , address=145.79.227.49:4367 , piece_id=CITJCIKXK4LYSUV6QRTLIHB7SEH5QOPUM7XR4JDHV54EJ5FLLMDQ, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.861538431 (551h48m43.732621466s) [noise=true , address=145.79.227.33:4904 , piece_id=4B64RHS2HQ6WZYND6MZGU4AZC4LQBXPNKOE2ABNVZQTEBXALQCDA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.863179095 (551h48m43.73426214s) [noise=true , address=77.90.31.19:4874 , piece_id=IXOOASSSDYJS42HWYZKK4PKE4LW47BT4WF5WRRXY7ANPLY632LRA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.864788100 (551h48m43.735871145s) [noise=true , address=77.90.31.16:4749 , piece_id=IVHKFK76TGLMJ7IBVS5ZCCMB35LKHF4FE5HLE7P5QAVGAWGSRHNA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.866240675 (551h48m43.73732371s) [noise=true , address=45.91.116.7:4029 , piece_id=VTKGXA2SWJZHOLLWREHQO66OHQZWHVRK25STFR62PEIJFGD4SG4A, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.867523768 (551h48m43.738606803s) [noise=true , address=45.91.116.6:4998 , piece_id=KCEYZDX2WGV7UCYX4H4BNSR2CWOHGPSEW76LKU2BGC3UDHA5M5SA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:45.143085883 (551h48m45.014168928s) [upload=483809]: Next returned nil segment Dec 28 15:17:45.143099779 (551h48m45.014182824s) [upload=483809]: Waiting for error group managing segments... Dec 28 15:17:45.472957433 (551h48m45.344040468s) [noise=true , address=64.49.13.139:4476 , piece_id=FUSCXA3KPS24G3R5ORJTDOUOU2KMIK5SNBPROHL2CRKMKBRWFN6A, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:45.739300305 (551h48m45.61038334s) [noise=true , address=145.79.227.6:4280 , piece_id=ARMYEAFMP6DPW2JSHWHOYPJ6OUUSUQETJCSSQQXFQX4DDCVZYLYQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:45.743884264 (551h48m45.614967309s) [noise=true , address=145.79.227.21:4508 , piece_id=PGAOV6UBGJPU7M5L3IR2LQU5LSISUJVKKBB7ZRV233F5MBEAWK6A, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:45.936004292 (551h48m45.807087328s) [noise=true , address=77.90.31.15:4727 , piece_id=YJDXXQTMZQRYCEAKLZUFPW6JO7RWHJMJAOFMURYFNLQSJ4LJG2CQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:45.968797306 (551h48m45.839880351s) [noise=true , address=77.90.31.16:4749 , piece_id=IVHKFK76TGLMJ7IBVS5ZCCMB35LKHF4FE5HLE7P5QAVGAWGSRHNA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.007515762 (551h48m45.878598797s) [noise=true , address=77.90.31.11:4580 , piece_id=UUPDJ75SL3HOJREC2SXJGQMUVLLP5WLN2L3LEON6KIKPB24DEKPQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.147397009 (551h48m46.018480054s) [noise=true , address=45.91.116.7:4029 , piece_id=VTKGXA2SWJZHOLLWREHQO66OHQZWHVRK25STFR62PEIJFGD4SG4A, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.460756007 (551h48m46.331839042s) [noise=true , address=193.149.164.143:4694 , piece_id=SDK5DIHWVFOZQIHQWJMZLOTQJEKRDBUTQ7R7ZTKOKF5LGECJRMOA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.662946081 (551h48m46.534029127s) [noise=true , address=145.79.227.43:4167 , piece_id=WEXP7LDX6O7KPXA6XIMRWNTMGNGSTE3XRFODPNMWFKR3D5WXX3LQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.695276604 (551h48m46.566359639s) [noise=true , address=45.91.116.6:4998 , piece_id=KCEYZDX2WGV7UCYX4H4BNSR2CWOHGPSEW76LKU2BGC3UDHA5M5SA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.739405984 (551h48m46.610489029s) [noise=true , address=145.79.227.33:4904 , piece_id=4B64RHS2HQ6WZYND6MZGU4AZC4LQBXPNKOE2ABNVZQTEBXALQCDA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.764293434 (551h48m46.635376479s) [noise=true , address=145.79.227.40:4083 , piece_id=RSMZDEGC4MX5TFRFKI33WNDK42NZDQ4M6UPRQWO4GE6VWGCJ6AFA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.771299524 (551h48m46.64238256s) [noise=true , address=145.79.227.30:4766 , piece_id=YA4HOOMXQRQCPE3VMQ6WEXBM3ZIED4RZLVFHZHRLBQJFQBFP2F7A, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.791263379 (551h48m46.662346414s) [noise=true , address=77.90.31.19:4874 , piece_id=IXOOASSSDYJS42HWYZKK4PKE4LW47BT4WF5WRRXY7ANPLY632LRA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.791282065 (551h48m46.6623651s) [seg_pos={0 0}, upload=483809]: Segment reached optimal threshold of 14 pieces. Dec 28 15:17:46.791451286 (551h48m46.662534321s) [noise=true , address=145.79.227.49:4367 , piece_id=CITJCIKXK4LYSUV6QRTLIHB7SEH5QOPUM7XR4JDHV54EJ5FLLMDQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF Dec 28 15:17:46.791458450 (551h48m46.662541496s) [noise=true , address=145.79.227.49:4367 , piece_id=CITJCIKXK4LYSUV6QRTLIHB7SEH5QOPUM7XR4JDHV54EJ5FLLMDQ, seg_pos={0 0}, upload=483809]: Long tail cancellation due to optimal threshold Dec 28 15:17:46.791491023 (551h48m46.662574058s) [noise=true , address=77.90.31.12:4610 , piece_id=RSI6EFWFTX4JZUBEFTV3VPMJWMWI7VDMUHG6DMK7CSKTH7MDFGXA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi): protocol: expected piece hash; context canceled; EOF Dec 28 15:17:46.791502935 (551h48m46.66258598s) [noise=true , address=77.90.31.12:4610 , piece_id=RSI6EFWFTX4JZUBEFTV3VPMJWMWI7VDMUHG6DMK7CSKTH7MDFGXA, seg_pos={0 0}, upload=483809]: Long tail cancellation due to optimal threshold Dec 28 15:17:46.791556076 (551h48m46.662639111s) [noise=true , address=145.79.227.34:4932 , piece_id=5NGLBPM6NRZIR4NJTA43KIFQQTD6DS6B37DIFCHXGT5YMDGNMXKA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qJfPZErJx4XGBw7NmfCLS7BogRnN29rUibuWkzQDFUNvPBpmm): protocol: expected piece hash; context canceled; EOF Dec 28 15:17:46.791561086 (551h48m46.662644131s) [noise=true , address=145.79.227.34:4932 , piece_id=5NGLBPM6NRZIR4NJTA43KIFQQTD6DS6B37DIFCHXGT5YMDGNMXKA, seg_pos={0 0}, upload=483809]: Long tail cancellation due to optimal threshold Dec 28 15:17:46.791659873 (551h48m46.662742908s) [noise=true , address=64.49.13.169:4716 , piece_id=V3PN2MUBRFNCLCYOVHYLXUORQO4S2TLRL6HMGJQCMJ6PCBZISPKQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:48986->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:48986->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48986->64.49.13.169:4716: use of closed network connection Dec 28 15:17:46.791665625 (551h48m46.66274866s) [noise=true , address=64.49.13.169:4716 , piece_id=V3PN2MUBRFNCLCYOVHYLXUORQO4S2TLRL6HMGJQCMJ6PCBZISPKQ, seg_pos={0 0}, upload=483809]: Long tail cancellation due to optimal threshold Dec 28 15:17:46.791739836 (551h48m46.662822881s) [seg_pos={0 0}, upload=483809]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 15:17:46.791808337 (551h48m46.662891372s) [upload=483809]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 15:17:46.791812865 (551h48m46.6628959s) [upload=483809]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 15:17:47.097154206 (551h48m46.968237251s) [upload=483809]: Done uploading segments... Dec 28 15:17:47.097173253 (551h48m46.968256288s) [upload=483809]: Upload finished. err: <nil> Duration: 40.409622637s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [X.X.........X.....X......X.X.X.....X......XX.X..XXX.X.XXX.XX] duration: 30.003995881s trace: https://jaeger.dev.storj.io/trace/56a60bfd907dd0ce FAIL (2025-12-28 15:27:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: XKWIUHSAUTEYYPNNRI6ZUE6Q2A7IU6FU4E5O26ZZY744I4URTVKQ) context deadline exceeded; piecestore: (Node ID: 129gPpguzPP8UDJEMDM9YPuKXiDVLhijvpeb6VYDMdzyccof35H, Piece ID: GIZKJMGYLBZ6V4EXP6G5SUVKJ5RU6HSOXIRG2PNGLTUXBNQNY3SA) context deadline exceeded ======LOGS====== Dec 28 15:27:20.752243379 (551h58m20.623326424s) [upload=483942]: Starting upload Dec 28 15:27:20.752314755 (551h58m20.62339781s) [upload=483942]: Done starting upload Dec 28 15:27:20.752395438 (551h58m20.623478473s) [upload=483942]: Uploading segments... Dec 28 15:27:20.752518202 (551h58m20.623601237s) [upload=483942]: Got next segment. Inline: false Dec 28 15:27:20.752554341 (551h58m20.623637386s) [upload=483942]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 15:27:20.752560202 (551h58m20.623643237s) [upload=483942]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 15:27:21.015134759 (551h58m20.886217804s) [seg_pos={0 0}, upload=483942]: Begin upload segment... Dec 28 15:27:21.015190095 (551h58m20.88627313s) [seg_pos={0 0}, upload=483942]: Done begin upload segment. Dec 28 15:27:21.015208369 (551h58m20.886291414s) [noise=true , address=145.79.227.38:4978 , piece_id=3G5ELCFYP626AAYGWBFGK7J5LMUUYJH3JAEY2BK3RIIUP2DFAWNA, seg_pos={0 0}, upload=483942]: Uploading piece... Dec 28 15:27:21.015223990 (551h58m20.886307025s) [noise=true , address=145.79.227.42:4121 , piece_id=YDYGBBEIFYTRDOVVD2AOGHVSK4CGGTXJD4EGGRSVHOLN7UGFO3NA, seg_pos={0 0}, upload=483942]: Uploading piece... Dec 28 15:27:21.016151024 (551h58m20.887234059s) [noise=true , address=77.90.31.21:4878 , piece_id=HO3LD7QLHWBKPOVGKHGTYZUKYQBL7VBOZKZDJZZW4FQ4N5EUMIEQ, seg_pos={0 0}, upload=483942]: Uploading piece... Dec 28 15:27:21.016595320 (551h58m20.887678365s) [noise=true , address=145.79.227.31:4798 , piece_id=NGUUZGXFKFAHUJLRQBPEY7B6VM5N4EDODLSXQWOYFNQKI6AYBOFQ, seg_pos={0 0}, upload=483942]: Uploading piece... Dec 28 15:27:21.016973921 (551h58m20.888056957s) [noise=true , address=145.79.227.50:4373 , piece_id=CLYDBUY6NO4E2KQG6PJNX6OIWYMQQ5GXF74KZGFGLDO4LC6DEKOQ, seg_pos={0 0}, upload=483942]: Uploading piece... Dec 28 15:27:21.017373893 (551h58m20.888456928s) [noise=true , address=77.90.31.14:4662 , piece_id=XKWIUHSAUTEYYPNNRI6ZUE6Q2A7IU6FU4E5O26ZZY744I4URTVKQ, seg_pos={0 0}, upload=483942]: Uploading piece... Dec 28 15:27:21.017786248 (551h58m20.888869283s) [noise=true , address=145.79.227.7:4294 , piece_id=KMGMXRNCANSOALFRLCIOKG2R4DCOSOGBIAGMFFSVR66BI3P5T7LQ, seg_pos={0 0}, upload=483942]: Uploading piece... Dec 28 15:27:21.018224803 (551h58m20.889307838s) [noise=true , address=145.79.227.4:4186 , piece_id=LY4WKJWSS3DMFCVYMXR5Q3GBFPGVFSHICCMEY4RNBGREHAFTGMLA, seg_pos={0 0}, upload=483942]: Uploading piece... Dec 28 15:27:21.019557892 (551h58m20.890640927s) [noise=true , address=77.90.31.18:4808 , piece_id=KUGXOQZZJFGD7TJCDVJ7ZBVSHWXHCHUPDZXN4RJASJLJYCKEV6AQ, seg_pos={0 0}, upload=483942]: Uploading piece... Dec 28 15:27:21.019767200 (551h58m20.890850235s) [noise=true , address=45.91.116.8:4046 , piece_id=GIZKJMGYLBZ6V4EXP6G5SUVKJ5RU6HSOXIRG2PNGLTUXBNQNY3SA, seg_pos={0 0}, upload=483942]: Uploading piece... Dec 28 15:27:21.020905919 (551h58m20.891988954s) [noise=true , address=45.91.116.14:4192 , piece_id=IDFMTTJGEPVIAQDONMWNBYDTURMPMZRQHJTHM3BJPPHBHF6GA42Q, seg_pos={0 0}, upload=483942]: Uploading piece... Dec 28 15:27:21.021227922 (551h58m20.892310967s) [noise=true , address=145.79.227.6:4258 , piece_id=GGQFGT44NK2L5T4H3Z3AWUNAXOZWUZXQJLL5NYCITNQNZ3DHOJSA, seg_pos={0 0}, upload=483942]: Uploading piece... Dec 28 15:27:21.022328337 (551h58m20.893411382s) [noise=true , address=145.79.227.14:4840 , piece_id=7JB5AK7OQBRGBLAA6YG6NMEEJJGQBH6EF3FFGZZFNVLCQU7ZP4UQ, seg_pos={0 0}, upload=483942]: Uploading piece... Dec 28 15:27:21.022693492 (551h58m20.893776527s) [noise=true , address=145.79.227.23:4546 , piece_id=UXUPM3THNJWF6WPZIB4PUQOLCXHJMHYMHS7DTQW4AYQZMVJKP2QQ, seg_pos={0 0}, upload=483942]: Uploading piece... Dec 28 15:27:21.023687896 (551h58m20.894770931s) [noise=true , address=145.79.227.17:4972 , piece_id=DFSNJUAMWENCCYTT4DNBLFB5I5SQG74HL5WSHC5TZ5IHAEYCBEZQ, seg_pos={0 0}, upload=483942]: Uploading piece... Dec 28 15:27:21.024042040 (551h58m20.895125075s) [noise=true , address=145.79.227.11:4730 , piece_id=OL23QJIWRUM6GDVANS5PZSXVF4WBFYYR7T5X7YQH2ZFWGOD7D5GA, seg_pos={0 0}, upload=483942]: Uploading piece... Dec 28 15:27:22.735609381 (551h58m22.606692416s) [upload=483942]: Next returned nil segment Dec 28 15:27:22.735628217 (551h58m22.606711252s) [upload=483942]: Waiting for error group managing segments... Dec 28 15:27:23.029257516 (551h58m22.900340551s) [noise=true , address=145.79.227.42:4121 , piece_id=YDYGBBEIFYTRDOVVD2AOGHVSK4CGGTXJD4EGGRSVHOLN7UGFO3NA, seg_pos={0 0}, upload=483942]: Done uploading piece. err: <nil> Dec 28 15:27:23.030924962 (551h58m22.902007997s) [noise=true , address=145.79.227.50:4373 , piece_id=CLYDBUY6NO4E2KQG6PJNX6OIWYMQQ5GXF74KZGFGLDO4LC6DEKOQ, seg_pos={0 0}, upload=483942]: Done uploading piece. err: <nil> Dec 28 15:27:23.033555650 (551h58m22.904638685s) [noise=true , address=145.79.227.23:4546 , piece_id=UXUPM3THNJWF6WPZIB4PUQOLCXHJMHYMHS7DTQW4AYQZMVJKP2QQ, seg_pos={0 0}, upload=483942]: Done uploading piece. err: <nil> Dec 28 15:27:23.036308322 (551h58m22.907391357s) [noise=true , address=145.79.227.31:4798 , piece_id=NGUUZGXFKFAHUJLRQBPEY7B6VM5N4EDODLSXQWOYFNQKI6AYBOFQ, seg_pos={0 0}, upload=483942]: Done uploading piece. err: <nil> Dec 28 15:27:23.036565863 (551h58m22.907648898s) [noise=true , address=145.79.227.38:4978 , piece_id=3G5ELCFYP626AAYGWBFGK7J5LMUUYJH3JAEY2BK3RIIUP2DFAWNA, seg_pos={0 0}, upload=483942]: Done uploading piece. err: <nil> Dec 28 15:27:23.040755891 (551h58m22.911838926s) [noise=true , address=145.79.227.17:4972 , piece_id=DFSNJUAMWENCCYTT4DNBLFB5I5SQG74HL5WSHC5TZ5IHAEYCBEZQ, seg_pos={0 0}, upload=483942]: Done uploading piece. err: <nil> Dec 28 15:27:23.044459984 (551h58m22.915543019s) [noise=true , address=145.79.227.6:4258 , piece_id=GGQFGT44NK2L5T4H3Z3AWUNAXOZWUZXQJLL5NYCITNQNZ3DHOJSA, seg_pos={0 0}, upload=483942]: Done uploading piece. err: <nil> Dec 28 15:27:23.044704419 (551h58m22.915787454s) [noise=true , address=145.79.227.14:4840 , piece_id=7JB5AK7OQBRGBLAA6YG6NMEEJJGQBH6EF3FFGZZFNVLCQU7ZP4UQ, seg_pos={0 0}, upload=483942]: Done uploading piece. err: <nil> Dec 28 15:27:23.050297249 (551h58m22.921380284s) [noise=true , address=145.79.227.7:4294 , piece_id=KMGMXRNCANSOALFRLCIOKG2R4DCOSOGBIAGMFFSVR66BI3P5T7LQ, seg_pos={0 0}, upload=483942]: Done uploading piece. err: <nil> Dec 28 15:27:23.052311224 (551h58m22.923394259s) [noise=true , address=145.79.227.4:4186 , piece_id=LY4WKJWSS3DMFCVYMXR5Q3GBFPGVFSHICCMEY4RNBGREHAFTGMLA, seg_pos={0 0}, upload=483942]: Done uploading piece. err: <nil> Dec 28 15:27:23.264045862 (551h58m23.135128897s) [noise=true , address=77.90.31.14:4662 , piece_id=XKWIUHSAUTEYYPNNRI6ZUE6Q2A7IU6FU4E5O26ZZY744I4URTVKQ, seg_pos={0 0}, upload=483942]: Done uploading piece. err: <nil> Dec 28 15:27:23.302920026 (551h58m23.174003061s) [noise=true , address=45.91.116.14:4192 , piece_id=IDFMTTJGEPVIAQDONMWNBYDTURMPMZRQHJTHM3BJPPHBHF6GA42Q, seg_pos={0 0}, upload=483942]: Done uploading piece. err: <nil> Dec 28 15:27:23.310694249 (551h58m23.181777284s) [noise=true , address=45.91.116.8:4046 , piece_id=GIZKJMGYLBZ6V4EXP6G5SUVKJ5RU6HSOXIRG2PNGLTUXBNQNY3SA, seg_pos={0 0}, upload=483942]: Done uploading piece. err: <nil> Dec 28 15:27:23.310715169 (551h58m23.181798204s) [seg_pos={0 0}, upload=483942]: Segment reached optimal threshold of 13 pieces. Dec 28 15:27:23.311021762 (551h58m23.182104807s) [noise=true , address=77.90.31.21:4878 , piece_id=HO3LD7QLHWBKPOVGKHGTYZUKYQBL7VBOZKZDJZZW4FQ4N5EUMIEQ, seg_pos={0 0}, upload=483942]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Dec 28 15:27:23.311040618 (551h58m23.182123653s) [noise=true , address=77.90.31.21:4878 , piece_id=HO3LD7QLHWBKPOVGKHGTYZUKYQBL7VBOZKZDJZZW4FQ4N5EUMIEQ, seg_pos={0 0}, upload=483942]: Long tail cancellation due to optimal threshold Dec 28 15:27:23.311065306 (551h58m23.182148351s) [noise=true , address=77.90.31.18:4808 , piece_id=KUGXOQZZJFGD7TJCDVJ7ZBVSHWXHCHUPDZXN4RJASJLJYCKEV6AQ, seg_pos={0 0}, upload=483942]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: read tcp 188.166.225.170:40802->77.90.31.18:4808: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 28 15:27:23.311072559 (551h58m23.182155594s) [noise=true , address=77.90.31.18:4808 , piece_id=KUGXOQZZJFGD7TJCDVJ7ZBVSHWXHCHUPDZXN4RJASJLJYCKEV6AQ, seg_pos={0 0}, upload=483942]: Long tail cancellation due to optimal threshold Dec 28 15:27:23.311249737 (551h58m23.182332772s) [noise=true , address=145.79.227.11:4730 , piece_id=OL23QJIWRUM6GDVANS5PZSXVF4WBFYYR7T5X7YQH2ZFWGOD7D5GA, seg_pos={0 0}, upload=483942]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom): protocol: expected piece hash; context canceled; EOF Dec 28 15:27:23.311258303 (551h58m23.182341338s) [noise=true , address=145.79.227.11:4730 , piece_id=OL23QJIWRUM6GDVANS5PZSXVF4WBFYYR7T5X7YQH2ZFWGOD7D5GA, seg_pos={0 0}, upload=483942]: Long tail cancellation due to optimal threshold Dec 28 15:27:23.311275536 (551h58m23.182358581s) [seg_pos={0 0}, upload=483942]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 15:27:23.311347533 (551h58m23.182430568s) [upload=483942]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 15:27:23.311352362 (551h58m23.182435407s) [upload=483942]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 15:27:23.600575321 (551h58m23.471658366s) [upload=483942]: Done uploading segments... Dec 28 15:27:23.600594657 (551h58m23.471677702s) [upload=483942]: Upload finished. err: <nil> Duration: 30.004008605s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [.X.X.........X.....X......X.X.X.....X......XX.X..XXX.X.XXX.X] duration: 30.002835569s trace: https://jaeger.dev.storj.io/trace/62febc8790db23ea FAIL (2025-12-28 15:27:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA, Piece ID: FAWSC4SAJX4DAPJFBBSG4GLUB6KQNTZCVRTSBP4VNDDXYZNOY2BA) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: TVQXAYREFGR65GNC37OSGAF4XTGKP6RVFCYVY2KSUFEJWD2FN7AA) context deadline exceeded ======LOGS====== Dec 28 15:26:50.533649971 (551h57m50.404733006s) [upload=483933]: Starting upload Dec 28 15:26:50.533737978 (551h57m50.404821013s) [upload=483933]: Done starting upload Dec 28 15:26:50.533781191 (551h57m50.404864226s) [upload=483933]: Uploading segments... Dec 28 15:26:50.534006911 (551h57m50.405089946s) [upload=483933]: Got next segment. Inline: false Dec 28 15:26:50.534039793 (551h57m50.405122838s) [upload=483933]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 15:26:50.534044592 (551h57m50.405127627s) [upload=483933]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 15:26:50.778659390 (551h57m50.649742425s) [seg_pos={0 0}, upload=483933]: Begin upload segment... Dec 28 15:26:50.778721058 (551h57m50.649804093s) [seg_pos={0 0}, upload=483933]: Done begin upload segment. Dec 28 15:26:50.778751265 (551h57m50.6498343s) [noise=true , address=145.79.227.41:4087 , piece_id=HDH4IKHQHTHQOKO7SGURA2N6MGFKMYFCEV5OXMSN3QT5BOSNCIUA, seg_pos={0 0}, upload=483933]: Uploading piece... Dec 28 15:26:50.778798475 (551h57m50.64988151s) [noise=true , address=77.90.31.14:4663 , piece_id=J3Y2V3AFAFEXNJ7PIYUD3KD6AQRETESTZZ2XDUYOIG4MXAOWQCKQ, seg_pos={0 0}, upload=483933]: Uploading piece... Dec 28 15:26:50.779224135 (551h57m50.65030717s) [noise=true , address=145.79.227.45:4229 , piece_id=MVVAIG7W67C3S2ALPXMC7XUAKVDOGGDKNLVEIYC4T7LNOTDYWJBA, seg_pos={0 0}, upload=483933]: Uploading piece... Dec 28 15:26:50.779604710 (551h57m50.650687745s) [noise=true , address=145.79.227.2:4113 , piece_id=U5PIWYSLQDTM22PWCGXTMJBZQMNJPPIIOYRGSPYJU7FKB2OYUBAA, seg_pos={0 0}, upload=483933]: Uploading piece... Dec 28 15:26:50.779920712 (551h57m50.651003757s) [noise=true , address=77.90.31.11:4553 , piece_id=HJYDJJTPCT63NT5FUM3YDJPOYFVQB2FFBNIJUTZ2SZHHNWEZQH6Q, seg_pos={0 0}, upload=483933]: Uploading piece... Dec 28 15:26:50.780227236 (551h57m50.651310271s) [noise=true , address=45.91.116.12:4228 , piece_id=FAWSC4SAJX4DAPJFBBSG4GLUB6KQNTZCVRTSBP4VNDDXYZNOY2BA, seg_pos={0 0}, upload=483933]: Uploading piece... Dec 28 15:26:50.780538278 (551h57m50.651621323s) [noise=true , address=45.91.116.5:4298 , piece_id=YWWQ5DHL6WEYHHPB425MGP7ITK5DEO22NEJTP3R235ISSS2BDSKQ, seg_pos={0 0}, upload=483933]: Uploading piece... Dec 28 15:26:50.780824372 (551h57m50.651907407s) [noise=true , address=145.79.227.16:4911 , piece_id=5DD45ERVMW6ZEJLCZJ5O3LDGEGEGRVHYLEZTGQIDDH2JQRGCEFBQ, seg_pos={0 0}, upload=483933]: Uploading piece... Dec 28 15:26:50.781161485 (551h57m50.65224453s) [noise=true , address=45.91.116.13:4156 , piece_id=TVQXAYREFGR65GNC37OSGAF4XTGKP6RVFCYVY2KSUFEJWD2FN7AA, seg_pos={0 0}, upload=483933]: Uploading piece... Dec 28 15:26:50.782181597 (551h57m50.653264642s) [noise=true , address=145.79.227.35:4482 , piece_id=GPUT32QQF45UVRMAXWRLQMO43F555OJRX5TZX3JOYSQIFIMYLI5Q, seg_pos={0 0}, upload=483933]: Uploading piece... Dec 28 15:26:50.783292112 (551h57m50.654375147s) [noise=true , address=77.90.31.21:4877 , piece_id=PEQR2F2QW6OM3EP6YIWT6Y5GGI3ETRL4FAAPKAQFZ5ZBSGARQXMQ, seg_pos={0 0}, upload=483933]: Uploading piece... Dec 28 15:26:50.785304253 (551h57m50.656387288s) [noise=true , address=145.79.227.42:4121 , piece_id=MXZKFDP344LLRKADQLWU72KSGOKVYINAL2ZCN6NVMAYVYOBD2CUA, seg_pos={0 0}, upload=483933]: Uploading piece... Dec 28 15:26:50.789114097 (551h57m50.660197142s) [noise=true , address=145.79.227.39:4015 , piece_id=ZQIFQERPKISYVGOS2E6I67H2OSFG2247OAW62XCIDTJ64WIANLNA, seg_pos={0 0}, upload=483933]: Uploading piece... Dec 28 15:26:50.791619308 (551h57m50.662702343s) [noise=true , address=145.79.227.34:4905 , piece_id=JI3ODHG4F2YRSTADHTFRMRVLZJ3AIUZMXTI7NADLET7JUTPUNTUQ, seg_pos={0 0}, upload=483933]: Uploading piece... Dec 28 15:26:50.792674136 (551h57m50.663757181s) [noise=true , address=145.79.227.17:4971 , piece_id=W7GP2TKTCGTSDJGOP3JRWHZDOT4JT5WQOW5DMUAHASFMA7XJCLFA, seg_pos={0 0}, upload=483933]: Uploading piece... Dec 28 15:26:50.794966732 (551h57m50.666049777s) [noise=true , address=145.79.227.50:4373 , piece_id=UYWVZCE5AQKGGHLUSURBALFH3YJMENOZWOKZYNQUTZHTMZVL2ZKA, seg_pos={0 0}, upload=483933]: Uploading piece... Dec 28 15:26:52.498336593 (551h57m52.369419628s) [upload=483933]: Next returned nil segment Dec 28 15:26:52.498369896 (551h57m52.369452931s) [upload=483933]: Waiting for error group managing segments... Dec 28 15:26:52.782710288 (551h57m52.653793373s) [noise=true , address=145.79.227.2:4113 , piece_id=U5PIWYSLQDTM22PWCGXTMJBZQMNJPPIIOYRGSPYJU7FKB2OYUBAA, seg_pos={0 0}, upload=483933]: Done uploading piece. err: <nil> Dec 28 15:26:52.786506365 (551h57m52.657589431s) [noise=true , address=145.79.227.16:4911 , piece_id=5DD45ERVMW6ZEJLCZJ5O3LDGEGEGRVHYLEZTGQIDDH2JQRGCEFBQ, seg_pos={0 0}, upload=483933]: Done uploading piece. err: <nil> Dec 28 15:26:52.786790337 (551h57m52.657873382s) [noise=true , address=145.79.227.45:4229 , piece_id=MVVAIG7W67C3S2ALPXMC7XUAKVDOGGDKNLVEIYC4T7LNOTDYWJBA, seg_pos={0 0}, upload=483933]: Done uploading piece. err: <nil> Dec 28 15:26:52.796863628 (551h57m52.667946673s) [noise=true , address=145.79.227.42:4121 , piece_id=MXZKFDP344LLRKADQLWU72KSGOKVYINAL2ZCN6NVMAYVYOBD2CUA, seg_pos={0 0}, upload=483933]: Done uploading piece. err: <nil> Dec 28 15:26:52.796936066 (551h57m52.668019101s) [noise=true , address=145.79.227.35:4482 , piece_id=GPUT32QQF45UVRMAXWRLQMO43F555OJRX5TZX3JOYSQIFIMYLI5Q, seg_pos={0 0}, upload=483933]: Done uploading piece. err: <nil> Dec 28 15:26:52.797017210 (551h57m52.668100245s) [noise=true , address=145.79.227.50:4373 , piece_id=UYWVZCE5AQKGGHLUSURBALFH3YJMENOZWOKZYNQUTZHTMZVL2ZKA, seg_pos={0 0}, upload=483933]: Done uploading piece. err: <nil> Dec 28 15:26:52.804972378 (551h57m52.676055414s) [noise=true , address=145.79.227.41:4087 , piece_id=HDH4IKHQHTHQOKO7SGURA2N6MGFKMYFCEV5OXMSN3QT5BOSNCIUA, seg_pos={0 0}, upload=483933]: Done uploading piece. err: <nil> Dec 28 15:26:52.817240469 (551h57m52.688323524s) [noise=true , address=145.79.227.39:4015 , piece_id=ZQIFQERPKISYVGOS2E6I67H2OSFG2247OAW62XCIDTJ64WIANLNA, seg_pos={0 0}, upload=483933]: Done uploading piece. err: <nil> Dec 28 15:26:52.817887571 (551h57m52.688970616s) [noise=true , address=145.79.227.17:4971 , piece_id=W7GP2TKTCGTSDJGOP3JRWHZDOT4JT5WQOW5DMUAHASFMA7XJCLFA, seg_pos={0 0}, upload=483933]: Done uploading piece. err: <nil> Dec 28 15:26:53.065038728 (551h57m52.936121763s) [noise=true , address=77.90.31.11:4553 , piece_id=HJYDJJTPCT63NT5FUM3YDJPOYFVQB2FFBNIJUTZ2SZHHNWEZQH6Q, seg_pos={0 0}, upload=483933]: Done uploading piece. err: <nil> Dec 28 15:26:53.099280320 (551h57m52.970363365s) [noise=true , address=77.90.31.14:4663 , piece_id=J3Y2V3AFAFEXNJ7PIYUD3KD6AQRETESTZZ2XDUYOIG4MXAOWQCKQ, seg_pos={0 0}, upload=483933]: Done uploading piece. err: <nil> Dec 28 15:26:53.390378670 (551h57m53.261461715s) [noise=true , address=45.91.116.13:4156 , piece_id=TVQXAYREFGR65GNC37OSGAF4XTGKP6RVFCYVY2KSUFEJWD2FN7AA, seg_pos={0 0}, upload=483933]: Done uploading piece. err: <nil> Dec 28 15:26:53.752707636 (551h57m53.623790671s) [noise=true , address=45.91.116.12:4228 , piece_id=FAWSC4SAJX4DAPJFBBSG4GLUB6KQNTZCVRTSBP4VNDDXYZNOY2BA, seg_pos={0 0}, upload=483933]: Done uploading piece. err: <nil> Dec 28 15:26:53.752731671 (551h57m53.623814706s) [seg_pos={0 0}, upload=483933]: Segment reached optimal threshold of 13 pieces. Dec 28 15:26:53.753105723 (551h57m53.624188758s) [noise=true , address=145.79.227.34:4905 , piece_id=JI3ODHG4F2YRSTADHTFRMRVLZJ3AIUZMXTI7NADLET7JUTPUNTUQ, seg_pos={0 0}, upload=483933]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 28 15:26:53.753129469 (551h57m53.624212534s) [noise=true , address=145.79.227.34:4905 , piece_id=JI3ODHG4F2YRSTADHTFRMRVLZJ3AIUZMXTI7NADLET7JUTPUNTUQ, seg_pos={0 0}, upload=483933]: Long tail cancellation due to optimal threshold Dec 28 15:26:53.753137935 (551h57m53.62422097s) [noise=true , address=45.91.116.5:4298 , piece_id=YWWQ5DHL6WEYHHPB425MGP7ITK5DEO22NEJTP3R235ISSS2BDSKQ, seg_pos={0 0}, upload=483933]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 28 15:26:53.753155217 (551h57m53.624238252s) [noise=true , address=45.91.116.5:4298 , piece_id=YWWQ5DHL6WEYHHPB425MGP7ITK5DEO22NEJTP3R235ISSS2BDSKQ, seg_pos={0 0}, upload=483933]: Long tail cancellation due to optimal threshold Dec 28 15:26:53.753330661 (551h57m53.624413696s) [noise=true , address=77.90.31.21:4877 , piece_id=PEQR2F2QW6OM3EP6YIWT6Y5GGI3ETRL4FAAPKAQFZ5ZBSGARQXMQ, seg_pos={0 0}, upload=483933]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12qy6swRPui25KAsuj9PzHvijxQFHoaLMJ95LvK34pCA9UU3gR2): protocol: expected piece hash; context canceled; EOF Dec 28 15:26:53.753337034 (551h57m53.624420069s) [noise=true , address=77.90.31.21:4877 , piece_id=PEQR2F2QW6OM3EP6YIWT6Y5GGI3ETRL4FAAPKAQFZ5ZBSGARQXMQ, seg_pos={0 0}, upload=483933]: Long tail cancellation due to optimal threshold Dec 28 15:26:53.753360668 (551h57m53.624443703s) [seg_pos={0 0}, upload=483933]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 15:26:53.753456331 (551h57m53.624539396s) [upload=483933]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 15:26:53.753461991 (551h57m53.624545026s) [upload=483933]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 15:26:54.056516648 (551h57m53.927599693s) [upload=483933]: Done uploading segments... Dec 28 15:26:54.056540173 (551h57m53.927623208s) [upload=483933]: Upload finished. err: <nil> Duration: 30.002856299s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 15.607881387s trace: https://jaeger.dev.storj.io/trace/4ce8174d0eaf2b7b FAIL (2025-12-28 15:27:13 UTC) orbiter warning: context deadline exceeded Duration: 15.60790906s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 40 out of 60 is failed (threshold=6) [.X..XXXXX.X.X.XXXXX...X.X.XX.XXX.X.XXXXX.XX.XX.X.XXXXX.X.XXX] duration: 5.001755731s trace: https://jaeger.dev.storj.io/trace/122d3f5514e3118f FAIL (2025-12-28 15:27:08 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001780517s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.X......X....X..........................X....X...X..........] duration: 43.243394927s trace: https://jaeger.dev.storj.io/trace/51e2216a4efa8b95 FAIL (2025-12-28 15:26:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 15s) [noise=true , address=145.79.227.40:4083 , piece_id=RSMZDEGC4MX5TFRFKI33WNDK42NZDQ4M6UPRQWO4GE6VWGCJ6AFA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.854006008 (551h48m43.725089043s) [noise=true , address=77.90.31.12:4610 , piece_id=RSI6EFWFTX4JZUBEFTV3VPMJWMWI7VDMUHG6DMK7CSKTH7MDFGXA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.854333892 (551h48m43.725416937s) [noise=true , address=64.49.13.139:4476 , piece_id=FUSCXA3KPS24G3R5ORJTDOUOU2KMIK5SNBPROHL2CRKMKBRWFN6A, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.854753131 (551h48m43.725836176s) [noise=true , address=145.79.227.34:4932 , piece_id=5NGLBPM6NRZIR4NJTA43KIFQQTD6DS6B37DIFCHXGT5YMDGNMXKA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.855102245 (551h48m43.726185381s) [noise=true , address=77.90.31.15:4727 , piece_id=YJDXXQTMZQRYCEAKLZUFPW6JO7RWHJMJAOFMURYFNLQSJ4LJG2CQ, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.855467040 (551h48m43.726550076s) [noise=true , address=145.79.227.6:4280 , piece_id=ARMYEAFMP6DPW2JSHWHOYPJ6OUUSUQETJCSSQQXFQX4DDCVZYLYQ, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.855834780 (551h48m43.726917815s) [noise=true , address=193.149.164.143:4694 , piece_id=SDK5DIHWVFOZQIHQWJMZLOTQJEKRDBUTQ7R7ZTKOKF5LGECJRMOA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.856263167 (551h48m43.727346212s) [noise=true , address=145.79.227.43:4167 , piece_id=WEXP7LDX6O7KPXA6XIMRWNTMGNGSTE3XRFODPNMWFKR3D5WXX3LQ, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.858002919 (551h48m43.729085964s) [noise=true , address=145.79.227.30:4766 , piece_id=YA4HOOMXQRQCPE3VMQ6WEXBM3ZIED4RZLVFHZHRLBQJFQBFP2F7A, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.858063876 (551h48m43.729146911s) [noise=true , address=64.49.13.169:4716 , piece_id=V3PN2MUBRFNCLCYOVHYLXUORQO4S2TLRL6HMGJQCMJ6PCBZISPKQ, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.859819067 (551h48m43.730902113s) [noise=true , address=145.79.227.49:4367 , piece_id=CITJCIKXK4LYSUV6QRTLIHB7SEH5QOPUM7XR4JDHV54EJ5FLLMDQ, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.861538431 (551h48m43.732621466s) [noise=true , address=145.79.227.33:4904 , piece_id=4B64RHS2HQ6WZYND6MZGU4AZC4LQBXPNKOE2ABNVZQTEBXALQCDA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.863179095 (551h48m43.73426214s) [noise=true , address=77.90.31.19:4874 , piece_id=IXOOASSSDYJS42HWYZKK4PKE4LW47BT4WF5WRRXY7ANPLY632LRA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.864788100 (551h48m43.735871145s) [noise=true , address=77.90.31.16:4749 , piece_id=IVHKFK76TGLMJ7IBVS5ZCCMB35LKHF4FE5HLE7P5QAVGAWGSRHNA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.866240675 (551h48m43.73732371s) [noise=true , address=45.91.116.7:4029 , piece_id=VTKGXA2SWJZHOLLWREHQO66OHQZWHVRK25STFR62PEIJFGD4SG4A, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.867523768 (551h48m43.738606803s) [noise=true , address=45.91.116.6:4998 , piece_id=KCEYZDX2WGV7UCYX4H4BNSR2CWOHGPSEW76LKU2BGC3UDHA5M5SA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:45.143085883 (551h48m45.014168928s) [upload=483809]: Next returned nil segment Dec 28 15:17:45.143099779 (551h48m45.014182824s) [upload=483809]: Waiting for error group managing segments... Dec 28 15:17:45.472957433 (551h48m45.344040468s) [noise=true , address=64.49.13.139:4476 , piece_id=FUSCXA3KPS24G3R5ORJTDOUOU2KMIK5SNBPROHL2CRKMKBRWFN6A, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:45.739300305 (551h48m45.61038334s) [noise=true , address=145.79.227.6:4280 , piece_id=ARMYEAFMP6DPW2JSHWHOYPJ6OUUSUQETJCSSQQXFQX4DDCVZYLYQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:45.743884264 (551h48m45.614967309s) [noise=true , address=145.79.227.21:4508 , piece_id=PGAOV6UBGJPU7M5L3IR2LQU5LSISUJVKKBB7ZRV233F5MBEAWK6A, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:45.936004292 (551h48m45.807087328s) [noise=true , address=77.90.31.15:4727 , piece_id=YJDXXQTMZQRYCEAKLZUFPW6JO7RWHJMJAOFMURYFNLQSJ4LJG2CQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:45.968797306 (551h48m45.839880351s) [noise=true , address=77.90.31.16:4749 , piece_id=IVHKFK76TGLMJ7IBVS5ZCCMB35LKHF4FE5HLE7P5QAVGAWGSRHNA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.007515762 (551h48m45.878598797s) [noise=true , address=77.90.31.11:4580 , piece_id=UUPDJ75SL3HOJREC2SXJGQMUVLLP5WLN2L3LEON6KIKPB24DEKPQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.147397009 (551h48m46.018480054s) [noise=true , address=45.91.116.7:4029 , piece_id=VTKGXA2SWJZHOLLWREHQO66OHQZWHVRK25STFR62PEIJFGD4SG4A, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.460756007 (551h48m46.331839042s) [noise=true , address=193.149.164.143:4694 , piece_id=SDK5DIHWVFOZQIHQWJMZLOTQJEKRDBUTQ7R7ZTKOKF5LGECJRMOA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.662946081 (551h48m46.534029127s) [noise=true , address=145.79.227.43:4167 , piece_id=WEXP7LDX6O7KPXA6XIMRWNTMGNGSTE3XRFODPNMWFKR3D5WXX3LQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.695276604 (551h48m46.566359639s) [noise=true , address=45.91.116.6:4998 , piece_id=KCEYZDX2WGV7UCYX4H4BNSR2CWOHGPSEW76LKU2BGC3UDHA5M5SA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.739405984 (551h48m46.610489029s) [noise=true , address=145.79.227.33:4904 , piece_id=4B64RHS2HQ6WZYND6MZGU4AZC4LQBXPNKOE2ABNVZQTEBXALQCDA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.764293434 (551h48m46.635376479s) [noise=true , address=145.79.227.40:4083 , piece_id=RSMZDEGC4MX5TFRFKI33WNDK42NZDQ4M6UPRQWO4GE6VWGCJ6AFA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.771299524 (551h48m46.64238256s) [noise=true , address=145.79.227.30:4766 , piece_id=YA4HOOMXQRQCPE3VMQ6WEXBM3ZIED4RZLVFHZHRLBQJFQBFP2F7A, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.791263379 (551h48m46.662346414s) [noise=true , address=77.90.31.19:4874 , piece_id=IXOOASSSDYJS42HWYZKK4PKE4LW47BT4WF5WRRXY7ANPLY632LRA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.791282065 (551h48m46.6623651s) [seg_pos={0 0}, upload=483809]: Segment reached optimal threshold of 14 pieces. Dec 28 15:17:46.791451286 (551h48m46.662534321s) [noise=true , address=145.79.227.49:4367 , piece_id=CITJCIKXK4LYSUV6QRTLIHB7SEH5QOPUM7XR4JDHV54EJ5FLLMDQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF Dec 28 15:17:46.791458450 (551h48m46.662541496s) [noise=true , address=145.79.227.49:4367 , piece_id=CITJCIKXK4LYSUV6QRTLIHB7SEH5QOPUM7XR4JDHV54EJ5FLLMDQ, seg_pos={0 0}, upload=483809]: Long tail cancellation due to optimal threshold Dec 28 15:17:46.791491023 (551h48m46.662574058s) [noise=true , address=77.90.31.12:4610 , piece_id=RSI6EFWFTX4JZUBEFTV3VPMJWMWI7VDMUHG6DMK7CSKTH7MDFGXA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi): protocol: expected piece hash; context canceled; EOF Dec 28 15:17:46.791502935 (551h48m46.66258598s) [noise=true , address=77.90.31.12:4610 , piece_id=RSI6EFWFTX4JZUBEFTV3VPMJWMWI7VDMUHG6DMK7CSKTH7MDFGXA, seg_pos={0 0}, upload=483809]: Long tail cancellation due to optimal threshold Dec 28 15:17:46.791556076 (551h48m46.662639111s) [noise=true , address=145.79.227.34:4932 , piece_id=5NGLBPM6NRZIR4NJTA43KIFQQTD6DS6B37DIFCHXGT5YMDGNMXKA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qJfPZErJx4XGBw7NmfCLS7BogRnN29rUibuWkzQDFUNvPBpmm): protocol: expected piece hash; context canceled; EOF Dec 28 15:17:46.791561086 (551h48m46.662644131s) [noise=true , address=145.79.227.34:4932 , piece_id=5NGLBPM6NRZIR4NJTA43KIFQQTD6DS6B37DIFCHXGT5YMDGNMXKA, seg_pos={0 0}, upload=483809]: Long tail cancellation due to optimal threshold Dec 28 15:17:46.791659873 (551h48m46.662742908s) [noise=true , address=64.49.13.169:4716 , piece_id=V3PN2MUBRFNCLCYOVHYLXUORQO4S2TLRL6HMGJQCMJ6PCBZISPKQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:48986->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:48986->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48986->64.49.13.169:4716: use of closed network connection Dec 28 15:17:46.791665625 (551h48m46.66274866s) [noise=true , address=64.49.13.169:4716 , piece_id=V3PN2MUBRFNCLCYOVHYLXUORQO4S2TLRL6HMGJQCMJ6PCBZISPKQ, seg_pos={0 0}, upload=483809]: Long tail cancellation due to optimal threshold Dec 28 15:17:46.791739836 (551h48m46.662822881s) [seg_pos={0 0}, upload=483809]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 15:17:46.791808337 (551h48m46.662891372s) [upload=483809]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 15:17:46.791812865 (551h48m46.6628959s) [upload=483809]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 15:17:47.097154206 (551h48m46.968237251s) [upload=483809]: Done uploading segments... Dec 28 15:17:47.097173253 (551h48m46.968256288s) [upload=483809]: Upload finished. err: <nil> Duration: 43.243419164s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 40 out of 60 is failed (threshold=6) [X.X..XXXXX.X.X.XXXXX...X.X.XX.XXX.X.XXXXX.XX.XX.X.XXXXX.X.XX] duration: 5.001960339s trace: https://jaeger.dev.storj.io/trace/42b4f4ab7db4ccea FAIL (2025-12-28 15:26:33 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001983343s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [..X.X.........X.....X......X.X.X.....X......XX.X..XXX.X.XXX.] duration: 6.610239401s trace: https://jaeger.dev.storj.io/trace/18fd8f8ad00ffea7 FAIL (2025-12-28 15:26:22 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: 6HZVMRK6BFKVROFFQWKYZKZU5ZVNJFD3J4R3HPMU4QFZJUP4HH3Q) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: MJXYJW5VRD6NVSYYVBUI4BRC65E4J5MSWD2KLMNMZFPRCLSBKRSA) context deadline exceeded ======LOGS====== Dec 28 15:25:42.923452222 (551h56m42.794535267s) [upload=483919]: Starting upload Dec 28 15:25:42.923526202 (551h56m42.794609247s) [upload=483919]: Done starting upload Dec 28 15:25:42.923613358 (551h56m42.794696393s) [upload=483919]: Uploading segments... Dec 28 15:25:42.923741362 (551h56m42.794824407s) [upload=483919]: Got next segment. Inline: false Dec 28 15:25:42.923771680 (551h56m42.794854725s) [upload=483919]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 15:25:42.923775958 (551h56m42.794858993s) [upload=483919]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 15:25:43.167662117 (551h56m43.038745152s) [seg_pos={0 0}, upload=483919]: Begin upload segment... Dec 28 15:25:43.167707323 (551h56m43.038790368s) [seg_pos={0 0}, upload=483919]: Done begin upload segment. Dec 28 15:25:43.167732761 (551h56m43.038815797s) [noise=true , address=145.79.227.7:4294 , piece_id=S3UTAEB5YZV5GCGSFMJJTVCZTRE5N2RK62T6TW6JHS7MCTXMJB3Q, seg_pos={0 0}, upload=483919]: Uploading piece... Dec 28 15:25:43.167853061 (551h56m43.038936096s) [noise=true , address=145.79.227.5:4221 , piece_id=DPEPKQSIYBBNP6KNVRQ22RUYT5TWSZFW2RXOKFO6MFBUPETZM67Q, seg_pos={0 0}, upload=483919]: Uploading piece... Dec 28 15:25:43.168472259 (551h56m43.039555294s) [noise=true , address=45.91.116.5:4300 , piece_id=6HZVMRK6BFKVROFFQWKYZKZU5ZVNJFD3J4R3HPMU4QFZJUP4HH3Q, seg_pos={0 0}, upload=483919]: Uploading piece... Dec 28 15:25:43.168778012 (551h56m43.039861047s) [noise=true , address=145.79.227.36:4519 , piece_id=3H3KRCWTF7B5D3IPRWFNZXVWNADV5I3V3ETSMMJB2UTNUYJVII4A, seg_pos={0 0}, upload=483919]: Uploading piece... Dec 28 15:25:43.169065669 (551h56m43.040148704s) [noise=true , address=45.91.116.12:4227 , piece_id=SHJPHG76XB3FFNJ72IOGTSPH3WWM2PIIIHTFFHJ6EBARXYOJZZXA, seg_pos={0 0}, upload=483919]: Uploading piece... Dec 28 15:25:43.169400567 (551h56m43.040483602s) [noise=true , address=145.79.227.17:4972 , piece_id=6MISLUZW2PA5UZQ4SYRVH7EYJZ6YOP2VTPMY4EC5O3CAJIG6DFVQ, seg_pos={0 0}, upload=483919]: Uploading piece... Dec 28 15:25:43.169731608 (551h56m43.040814643s) [noise=true , address=145.79.227.47:4266 , piece_id=2CTKGFOK44WLRMSOPWNKMDEX4TA445YZF75CIB2QCM4ZVJ3MT7HQ, seg_pos={0 0}, upload=483919]: Uploading piece... Dec 28 15:25:43.170095911 (551h56m43.041178946s) [noise=true , address=45.91.116.6:4976 , piece_id=JIXO2H3CROJPYNWHTKQZCPH4A5HXNEW2Q5AO2GGUFAEEYJDHXLYA, seg_pos={0 0}, upload=483919]: Uploading piece... Dec 28 15:25:43.170338584 (551h56m43.041421629s) [noise=true , address=77.90.31.13:4626 , piece_id=5FBLESMHLZ7ZSZCF6C5SFG32RCUUHF75UDNSXFJMXQLGVNRR5XTA, seg_pos={0 0}, upload=483919]: Uploading piece... Dec 28 15:25:43.170582217 (551h56m43.041665253s) [noise=true , address=145.79.227.6:4257 , piece_id=DPC4BK76BYHTJ7HZAMU6K5JYS3ZQULAM7Z2ZIFQM7FOMVI3QSOMQ, seg_pos={0 0}, upload=483919]: Uploading piece... Dec 28 15:25:43.170901726 (551h56m43.041984761s) [noise=true , address=145.79.227.15:4876 , piece_id=TRMRTYGRBPD5YFSK4XVEQDKSVOLC35P3AXJJNPKLSG7BGWQMCMFQ, seg_pos={0 0}, upload=483919]: Uploading piece... Dec 28 15:25:43.171412218 (551h56m43.042495263s) [noise=true , address=145.79.227.42:4121 , piece_id=OIWH5E2SNGCM6ELSVW23EXHMW4WQ7Z5SEEYVWF56CXK33HD2Y3EA, seg_pos={0 0}, upload=483919]: Uploading piece... Dec 28 15:25:43.171996771 (551h56m43.043079816s) [noise=true , address=45.91.116.14:4189 , piece_id=UHSYJL4PGMPWBWL42SLJBAYDPGILJH4JNNE6SQGARIDCWE5PJMDA, seg_pos={0 0}, upload=483919]: Uploading piece... Dec 28 15:25:43.172425137 (551h56m43.043508182s) [noise=true , address=45.91.116.13:4156 , piece_id=MJXYJW5VRD6NVSYYVBUI4BRC65E4J5MSWD2KLMNMZFPRCLSBKRSA, seg_pos={0 0}, upload=483919]: Uploading piece... Dec 28 15:25:43.172449734 (551h56m43.043532769s) [noise=true , address=145.79.227.29:4943 , piece_id=MROJM2HGY57QX4X3J5W3HHOVC2CR67QVT5MW2UO4QZO3DJAJCYYA, seg_pos={0 0}, upload=483919]: Uploading piece... Dec 28 15:25:43.172841580 (551h56m43.043924615s) [noise=true , address=145.79.227.16:4912 , piece_id=AKR2XHNT4MTJKDGF3FGKSPRXVY5XYSOGT6S27GFJI5VBHHJP4JXA, seg_pos={0 0}, upload=483919]: Uploading piece... Dec 28 15:25:44.878333864 (551h56m44.749416909s) [upload=483919]: Next returned nil segment Dec 28 15:25:44.878349934 (551h56m44.749432979s) [upload=483919]: Waiting for error group managing segments... Dec 28 15:25:45.162940332 (551h56m45.034023367s) [noise=true , address=145.79.227.15:4876 , piece_id=TRMRTYGRBPD5YFSK4XVEQDKSVOLC35P3AXJJNPKLSG7BGWQMCMFQ, seg_pos={0 0}, upload=483919]: Done uploading piece. err: <nil> Dec 28 15:25:45.170145312 (551h56m45.041228347s) [noise=true , address=145.79.227.7:4294 , piece_id=S3UTAEB5YZV5GCGSFMJJTVCZTRE5N2RK62T6TW6JHS7MCTXMJB3Q, seg_pos={0 0}, upload=483919]: Done uploading piece. err: <nil> Dec 28 15:25:45.175492884 (551h56m45.046575919s) [noise=true , address=145.79.227.36:4519 , piece_id=3H3KRCWTF7B5D3IPRWFNZXVWNADV5I3V3ETSMMJB2UTNUYJVII4A, seg_pos={0 0}, upload=483919]: Done uploading piece. err: <nil> Dec 28 15:25:45.175864802 (551h56m45.046947837s) [noise=true , address=145.79.227.17:4972 , piece_id=6MISLUZW2PA5UZQ4SYRVH7EYJZ6YOP2VTPMY4EC5O3CAJIG6DFVQ, seg_pos={0 0}, upload=483919]: Done uploading piece. err: <nil> Dec 28 15:25:45.177831899 (551h56m45.048914944s) [noise=true , address=145.79.227.16:4912 , piece_id=AKR2XHNT4MTJKDGF3FGKSPRXVY5XYSOGT6S27GFJI5VBHHJP4JXA, seg_pos={0 0}, upload=483919]: Done uploading piece. err: <nil> Dec 28 15:25:45.178579712 (551h56m45.049662757s) [noise=true , address=145.79.227.6:4257 , piece_id=DPC4BK76BYHTJ7HZAMU6K5JYS3ZQULAM7Z2ZIFQM7FOMVI3QSOMQ, seg_pos={0 0}, upload=483919]: Done uploading piece. err: <nil> Dec 28 15:25:45.187389257 (551h56m45.058472292s) [noise=true , address=145.79.227.47:4266 , piece_id=2CTKGFOK44WLRMSOPWNKMDEX4TA445YZF75CIB2QCM4ZVJ3MT7HQ, seg_pos={0 0}, upload=483919]: Done uploading piece. err: <nil> Dec 28 15:25:45.193433627 (551h56m45.064516672s) [noise=true , address=145.79.227.5:4221 , piece_id=DPEPKQSIYBBNP6KNVRQ22RUYT5TWSZFW2RXOKFO6MFBUPETZM67Q, seg_pos={0 0}, upload=483919]: Done uploading piece. err: <nil> Dec 28 15:25:45.198005883 (551h56m45.069088928s) [noise=true , address=145.79.227.42:4121 , piece_id=OIWH5E2SNGCM6ELSVW23EXHMW4WQ7Z5SEEYVWF56CXK33HD2Y3EA, seg_pos={0 0}, upload=483919]: Done uploading piece. err: <nil> Dec 28 15:25:45.389273328 (551h56m45.260356373s) [noise=true , address=77.90.31.13:4626 , piece_id=5FBLESMHLZ7ZSZCF6C5SFG32RCUUHF75UDNSXFJMXQLGVNRR5XTA, seg_pos={0 0}, upload=483919]: Done uploading piece. err: <nil> Dec 28 15:25:45.493447122 (551h56m45.364530157s) [noise=true , address=45.91.116.12:4227 , piece_id=SHJPHG76XB3FFNJ72IOGTSPH3WWM2PIIIHTFFHJ6EBARXYOJZZXA, seg_pos={0 0}, upload=483919]: Done uploading piece. err: <nil> Dec 28 15:25:45.566113711 (551h56m45.437196756s) [noise=true , address=45.91.116.5:4300 , piece_id=6HZVMRK6BFKVROFFQWKYZKZU5ZVNJFD3J4R3HPMU4QFZJUP4HH3Q, seg_pos={0 0}, upload=483919]: Done uploading piece. err: <nil> Dec 28 15:25:45.665931000 (551h56m45.537014045s) [noise=true , address=45.91.116.13:4156 , piece_id=MJXYJW5VRD6NVSYYVBUI4BRC65E4J5MSWD2KLMNMZFPRCLSBKRSA, seg_pos={0 0}, upload=483919]: Done uploading piece. err: <nil> Dec 28 15:25:45.665951489 (551h56m45.537034535s) [seg_pos={0 0}, upload=483919]: Segment reached optimal threshold of 13 pieces. Dec 28 15:25:45.666146851 (551h56m45.537229886s) [noise=true , address=145.79.227.29:4943 , piece_id=MROJM2HGY57QX4X3J5W3HHOVC2CR67QVT5MW2UO4QZO3DJAJCYYA, seg_pos={0 0}, upload=483919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz): protocol: expected piece hash; context canceled; EOF Dec 28 15:25:45.666153403 (551h56m45.537236438s) [noise=true , address=145.79.227.29:4943 , piece_id=MROJM2HGY57QX4X3J5W3HHOVC2CR67QVT5MW2UO4QZO3DJAJCYYA, seg_pos={0 0}, upload=483919]: Long tail cancellation due to optimal threshold Dec 28 15:25:45.666270677 (551h56m45.537353712s) [noise=true , address=45.91.116.6:4976 , piece_id=JIXO2H3CROJPYNWHTKQZCPH4A5HXNEW2Q5AO2GGUFAEEYJDHXLYA, seg_pos={0 0}, upload=483919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 28 15:25:45.666286537 (551h56m45.537369582s) [noise=true , address=45.91.116.6:4976 , piece_id=JIXO2H3CROJPYNWHTKQZCPH4A5HXNEW2Q5AO2GGUFAEEYJDHXLYA, seg_pos={0 0}, upload=483919]: Long tail cancellation due to optimal threshold Dec 28 15:25:45.666302057 (551h56m45.537385102s) [noise=true , address=45.91.116.14:4189 , piece_id=UHSYJL4PGMPWBWL42SLJBAYDPGILJH4JNNE6SQGARIDCWE5PJMDA, seg_pos={0 0}, upload=483919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 28 15:25:45.666311645 (551h56m45.53739469s) [noise=true , address=45.91.116.14:4189 , piece_id=UHSYJL4PGMPWBWL42SLJBAYDPGILJH4JNNE6SQGARIDCWE5PJMDA, seg_pos={0 0}, upload=483919]: Long tail cancellation due to optimal threshold Dec 28 15:25:45.666336002 (551h56m45.537419047s) [seg_pos={0 0}, upload=483919]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 15:25:45.666515413 (551h56m45.537598458s) [upload=483919]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 15:25:45.666525813 (551h56m45.537608858s) [upload=483919]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 15:25:45.959655951 (551h56m45.830738986s) [upload=483919]: Done uploading segments... Dec 28 15:25:45.959678865 (551h56m45.83076191s) [upload=483919]: Upload finished. err: <nil> Duration: 6.610269458s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 16.959711417s trace: https://jaeger.dev.storj.io/trace/4ce8174d0eaf2b7b FAIL (2025-12-28 15:26:13 UTC) orbiter warning: context deadline exceeded Duration: 16.959743459s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [...X.X.........X.....X......X.X.X.....X......XX.X..XXX.X.XXX] duration: 30.003341023s trace: https://jaeger.dev.storj.io/trace/18fd8f8ad00ffea7 FAIL (2025-12-28 15:26:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: 6HZVMRK6BFKVROFFQWKYZKZU5ZVNJFD3J4R3HPMU4QFZJUP4HH3Q) context deadline exceeded; piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: MJXYJW5VRD6NVSYYVBUI4BRC65E4J5MSWD2KLMNMZFPRCLSBKRSA) context deadline exceeded ======LOGS====== Dec 28 15:25:42.923452222 (551h56m42.794535267s) [upload=483919]: Starting upload Dec 28 15:25:42.923526202 (551h56m42.794609247s) [upload=483919]: Done starting upload Dec 28 15:25:42.923613358 (551h56m42.794696393s) [upload=483919]: Uploading segments... Dec 28 15:25:42.923741362 (551h56m42.794824407s) [upload=483919]: Got next segment. Inline: false Dec 28 15:25:42.923771680 (551h56m42.794854725s) [upload=483919]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 15:25:42.923775958 (551h56m42.794858993s) [upload=483919]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 15:25:43.167662117 (551h56m43.038745152s) [seg_pos={0 0}, upload=483919]: Begin upload segment... Dec 28 15:25:43.167707323 (551h56m43.038790368s) [seg_pos={0 0}, upload=483919]: Done begin upload segment. Dec 28 15:25:43.167732761 (551h56m43.038815797s) [noise=true , address=145.79.227.7:4294 , piece_id=S3UTAEB5YZV5GCGSFMJJTVCZTRE5N2RK62T6TW6JHS7MCTXMJB3Q, seg_pos={0 0}, upload=483919]: Uploading piece... Dec 28 15:25:43.167853061 (551h56m43.038936096s) [noise=true , address=145.79.227.5:4221 , piece_id=DPEPKQSIYBBNP6KNVRQ22RUYT5TWSZFW2RXOKFO6MFBUPETZM67Q, seg_pos={0 0}, upload=483919]: Uploading piece... Dec 28 15:25:43.168472259 (551h56m43.039555294s) [noise=true , address=45.91.116.5:4300 , piece_id=6HZVMRK6BFKVROFFQWKYZKZU5ZVNJFD3J4R3HPMU4QFZJUP4HH3Q, seg_pos={0 0}, upload=483919]: Uploading piece... Dec 28 15:25:43.168778012 (551h56m43.039861047s) [noise=true , address=145.79.227.36:4519 , piece_id=3H3KRCWTF7B5D3IPRWFNZXVWNADV5I3V3ETSMMJB2UTNUYJVII4A, seg_pos={0 0}, upload=483919]: Uploading piece... Dec 28 15:25:43.169065669 (551h56m43.040148704s) [noise=true , address=45.91.116.12:4227 , piece_id=SHJPHG76XB3FFNJ72IOGTSPH3WWM2PIIIHTFFHJ6EBARXYOJZZXA, seg_pos={0 0}, upload=483919]: Uploading piece... Dec 28 15:25:43.169400567 (551h56m43.040483602s) [noise=true , address=145.79.227.17:4972 , piece_id=6MISLUZW2PA5UZQ4SYRVH7EYJZ6YOP2VTPMY4EC5O3CAJIG6DFVQ, seg_pos={0 0}, upload=483919]: Uploading piece... Dec 28 15:25:43.169731608 (551h56m43.040814643s) [noise=true , address=145.79.227.47:4266 , piece_id=2CTKGFOK44WLRMSOPWNKMDEX4TA445YZF75CIB2QCM4ZVJ3MT7HQ, seg_pos={0 0}, upload=483919]: Uploading piece... Dec 28 15:25:43.170095911 (551h56m43.041178946s) [noise=true , address=45.91.116.6:4976 , piece_id=JIXO2H3CROJPYNWHTKQZCPH4A5HXNEW2Q5AO2GGUFAEEYJDHXLYA, seg_pos={0 0}, upload=483919]: Uploading piece... Dec 28 15:25:43.170338584 (551h56m43.041421629s) [noise=true , address=77.90.31.13:4626 , piece_id=5FBLESMHLZ7ZSZCF6C5SFG32RCUUHF75UDNSXFJMXQLGVNRR5XTA, seg_pos={0 0}, upload=483919]: Uploading piece... Dec 28 15:25:43.170582217 (551h56m43.041665253s) [noise=true , address=145.79.227.6:4257 , piece_id=DPC4BK76BYHTJ7HZAMU6K5JYS3ZQULAM7Z2ZIFQM7FOMVI3QSOMQ, seg_pos={0 0}, upload=483919]: Uploading piece... Dec 28 15:25:43.170901726 (551h56m43.041984761s) [noise=true , address=145.79.227.15:4876 , piece_id=TRMRTYGRBPD5YFSK4XVEQDKSVOLC35P3AXJJNPKLSG7BGWQMCMFQ, seg_pos={0 0}, upload=483919]: Uploading piece... Dec 28 15:25:43.171412218 (551h56m43.042495263s) [noise=true , address=145.79.227.42:4121 , piece_id=OIWH5E2SNGCM6ELSVW23EXHMW4WQ7Z5SEEYVWF56CXK33HD2Y3EA, seg_pos={0 0}, upload=483919]: Uploading piece... Dec 28 15:25:43.171996771 (551h56m43.043079816s) [noise=true , address=45.91.116.14:4189 , piece_id=UHSYJL4PGMPWBWL42SLJBAYDPGILJH4JNNE6SQGARIDCWE5PJMDA, seg_pos={0 0}, upload=483919]: Uploading piece... Dec 28 15:25:43.172425137 (551h56m43.043508182s) [noise=true , address=45.91.116.13:4156 , piece_id=MJXYJW5VRD6NVSYYVBUI4BRC65E4J5MSWD2KLMNMZFPRCLSBKRSA, seg_pos={0 0}, upload=483919]: Uploading piece... Dec 28 15:25:43.172449734 (551h56m43.043532769s) [noise=true , address=145.79.227.29:4943 , piece_id=MROJM2HGY57QX4X3J5W3HHOVC2CR67QVT5MW2UO4QZO3DJAJCYYA, seg_pos={0 0}, upload=483919]: Uploading piece... Dec 28 15:25:43.172841580 (551h56m43.043924615s) [noise=true , address=145.79.227.16:4912 , piece_id=AKR2XHNT4MTJKDGF3FGKSPRXVY5XYSOGT6S27GFJI5VBHHJP4JXA, seg_pos={0 0}, upload=483919]: Uploading piece... Dec 28 15:25:44.878333864 (551h56m44.749416909s) [upload=483919]: Next returned nil segment Dec 28 15:25:44.878349934 (551h56m44.749432979s) [upload=483919]: Waiting for error group managing segments... Dec 28 15:25:45.162940332 (551h56m45.034023367s) [noise=true , address=145.79.227.15:4876 , piece_id=TRMRTYGRBPD5YFSK4XVEQDKSVOLC35P3AXJJNPKLSG7BGWQMCMFQ, seg_pos={0 0}, upload=483919]: Done uploading piece. err: <nil> Dec 28 15:25:45.170145312 (551h56m45.041228347s) [noise=true , address=145.79.227.7:4294 , piece_id=S3UTAEB5YZV5GCGSFMJJTVCZTRE5N2RK62T6TW6JHS7MCTXMJB3Q, seg_pos={0 0}, upload=483919]: Done uploading piece. err: <nil> Dec 28 15:25:45.175492884 (551h56m45.046575919s) [noise=true , address=145.79.227.36:4519 , piece_id=3H3KRCWTF7B5D3IPRWFNZXVWNADV5I3V3ETSMMJB2UTNUYJVII4A, seg_pos={0 0}, upload=483919]: Done uploading piece. err: <nil> Dec 28 15:25:45.175864802 (551h56m45.046947837s) [noise=true , address=145.79.227.17:4972 , piece_id=6MISLUZW2PA5UZQ4SYRVH7EYJZ6YOP2VTPMY4EC5O3CAJIG6DFVQ, seg_pos={0 0}, upload=483919]: Done uploading piece. err: <nil> Dec 28 15:25:45.177831899 (551h56m45.048914944s) [noise=true , address=145.79.227.16:4912 , piece_id=AKR2XHNT4MTJKDGF3FGKSPRXVY5XYSOGT6S27GFJI5VBHHJP4JXA, seg_pos={0 0}, upload=483919]: Done uploading piece. err: <nil> Dec 28 15:25:45.178579712 (551h56m45.049662757s) [noise=true , address=145.79.227.6:4257 , piece_id=DPC4BK76BYHTJ7HZAMU6K5JYS3ZQULAM7Z2ZIFQM7FOMVI3QSOMQ, seg_pos={0 0}, upload=483919]: Done uploading piece. err: <nil> Dec 28 15:25:45.187389257 (551h56m45.058472292s) [noise=true , address=145.79.227.47:4266 , piece_id=2CTKGFOK44WLRMSOPWNKMDEX4TA445YZF75CIB2QCM4ZVJ3MT7HQ, seg_pos={0 0}, upload=483919]: Done uploading piece. err: <nil> Dec 28 15:25:45.193433627 (551h56m45.064516672s) [noise=true , address=145.79.227.5:4221 , piece_id=DPEPKQSIYBBNP6KNVRQ22RUYT5TWSZFW2RXOKFO6MFBUPETZM67Q, seg_pos={0 0}, upload=483919]: Done uploading piece. err: <nil> Dec 28 15:25:45.198005883 (551h56m45.069088928s) [noise=true , address=145.79.227.42:4121 , piece_id=OIWH5E2SNGCM6ELSVW23EXHMW4WQ7Z5SEEYVWF56CXK33HD2Y3EA, seg_pos={0 0}, upload=483919]: Done uploading piece. err: <nil> Dec 28 15:25:45.389273328 (551h56m45.260356373s) [noise=true , address=77.90.31.13:4626 , piece_id=5FBLESMHLZ7ZSZCF6C5SFG32RCUUHF75UDNSXFJMXQLGVNRR5XTA, seg_pos={0 0}, upload=483919]: Done uploading piece. err: <nil> Dec 28 15:25:45.493447122 (551h56m45.364530157s) [noise=true , address=45.91.116.12:4227 , piece_id=SHJPHG76XB3FFNJ72IOGTSPH3WWM2PIIIHTFFHJ6EBARXYOJZZXA, seg_pos={0 0}, upload=483919]: Done uploading piece. err: <nil> Dec 28 15:25:45.566113711 (551h56m45.437196756s) [noise=true , address=45.91.116.5:4300 , piece_id=6HZVMRK6BFKVROFFQWKYZKZU5ZVNJFD3J4R3HPMU4QFZJUP4HH3Q, seg_pos={0 0}, upload=483919]: Done uploading piece. err: <nil> Dec 28 15:25:45.665931000 (551h56m45.537014045s) [noise=true , address=45.91.116.13:4156 , piece_id=MJXYJW5VRD6NVSYYVBUI4BRC65E4J5MSWD2KLMNMZFPRCLSBKRSA, seg_pos={0 0}, upload=483919]: Done uploading piece. err: <nil> Dec 28 15:25:45.665951489 (551h56m45.537034535s) [seg_pos={0 0}, upload=483919]: Segment reached optimal threshold of 13 pieces. Dec 28 15:25:45.666146851 (551h56m45.537229886s) [noise=true , address=145.79.227.29:4943 , piece_id=MROJM2HGY57QX4X3J5W3HHOVC2CR67QVT5MW2UO4QZO3DJAJCYYA, seg_pos={0 0}, upload=483919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12fc5Trj8XxacZhzxdksUVT5CbG6uTmiwdUPjreYi4UMPY1TXWz): protocol: expected piece hash; context canceled; EOF Dec 28 15:25:45.666153403 (551h56m45.537236438s) [noise=true , address=145.79.227.29:4943 , piece_id=MROJM2HGY57QX4X3J5W3HHOVC2CR67QVT5MW2UO4QZO3DJAJCYYA, seg_pos={0 0}, upload=483919]: Long tail cancellation due to optimal threshold Dec 28 15:25:45.666270677 (551h56m45.537353712s) [noise=true , address=45.91.116.6:4976 , piece_id=JIXO2H3CROJPYNWHTKQZCPH4A5HXNEW2Q5AO2GGUFAEEYJDHXLYA, seg_pos={0 0}, upload=483919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp): protocol: expected piece hash; context canceled; EOF Dec 28 15:25:45.666286537 (551h56m45.537369582s) [noise=true , address=45.91.116.6:4976 , piece_id=JIXO2H3CROJPYNWHTKQZCPH4A5HXNEW2Q5AO2GGUFAEEYJDHXLYA, seg_pos={0 0}, upload=483919]: Long tail cancellation due to optimal threshold Dec 28 15:25:45.666302057 (551h56m45.537385102s) [noise=true , address=45.91.116.14:4189 , piece_id=UHSYJL4PGMPWBWL42SLJBAYDPGILJH4JNNE6SQGARIDCWE5PJMDA, seg_pos={0 0}, upload=483919]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: expected piece hash; context canceled; EOF Dec 28 15:25:45.666311645 (551h56m45.53739469s) [noise=true , address=45.91.116.14:4189 , piece_id=UHSYJL4PGMPWBWL42SLJBAYDPGILJH4JNNE6SQGARIDCWE5PJMDA, seg_pos={0 0}, upload=483919]: Long tail cancellation due to optimal threshold Dec 28 15:25:45.666336002 (551h56m45.537419047s) [seg_pos={0 0}, upload=483919]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 15:25:45.666515413 (551h56m45.537598458s) [upload=483919]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 15:25:45.666525813 (551h56m45.537608858s) [upload=483919]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 15:25:45.959655951 (551h56m45.830738986s) [upload=483919]: Done uploading segments... Dec 28 15:25:45.959678865 (551h56m45.83076191s) [upload=483919]: Upload finished. err: <nil> Duration: 30.003362014s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 40 out of 60 is failed (threshold=6) [XX.X..XXXXX.X.X.XXXXX...X.X.XX.XXX.X.XXXXX.XX.XX.X.XXXXX.X.X] duration: 5.001266779s trace: https://jaeger.dev.storj.io/trace/37eb963460e96265 FAIL (2025-12-28 15:26:11 UTC) orbiter warning: context deadline exceeded Duration: 5.001285053s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [..X......X....X..........................X....X...X.........] duration: 43.222979935s trace: https://jaeger.dev.storj.io/trace/51e2216a4efa8b95 FAIL (2025-12-28 15:25:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 15s) [noise=true , address=145.79.227.40:4083 , piece_id=RSMZDEGC4MX5TFRFKI33WNDK42NZDQ4M6UPRQWO4GE6VWGCJ6AFA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.854006008 (551h48m43.725089043s) [noise=true , address=77.90.31.12:4610 , piece_id=RSI6EFWFTX4JZUBEFTV3VPMJWMWI7VDMUHG6DMK7CSKTH7MDFGXA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.854333892 (551h48m43.725416937s) [noise=true , address=64.49.13.139:4476 , piece_id=FUSCXA3KPS24G3R5ORJTDOUOU2KMIK5SNBPROHL2CRKMKBRWFN6A, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.854753131 (551h48m43.725836176s) [noise=true , address=145.79.227.34:4932 , piece_id=5NGLBPM6NRZIR4NJTA43KIFQQTD6DS6B37DIFCHXGT5YMDGNMXKA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.855102245 (551h48m43.726185381s) [noise=true , address=77.90.31.15:4727 , piece_id=YJDXXQTMZQRYCEAKLZUFPW6JO7RWHJMJAOFMURYFNLQSJ4LJG2CQ, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.855467040 (551h48m43.726550076s) [noise=true , address=145.79.227.6:4280 , piece_id=ARMYEAFMP6DPW2JSHWHOYPJ6OUUSUQETJCSSQQXFQX4DDCVZYLYQ, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.855834780 (551h48m43.726917815s) [noise=true , address=193.149.164.143:4694 , piece_id=SDK5DIHWVFOZQIHQWJMZLOTQJEKRDBUTQ7R7ZTKOKF5LGECJRMOA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.856263167 (551h48m43.727346212s) [noise=true , address=145.79.227.43:4167 , piece_id=WEXP7LDX6O7KPXA6XIMRWNTMGNGSTE3XRFODPNMWFKR3D5WXX3LQ, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.858002919 (551h48m43.729085964s) [noise=true , address=145.79.227.30:4766 , piece_id=YA4HOOMXQRQCPE3VMQ6WEXBM3ZIED4RZLVFHZHRLBQJFQBFP2F7A, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.858063876 (551h48m43.729146911s) [noise=true , address=64.49.13.169:4716 , piece_id=V3PN2MUBRFNCLCYOVHYLXUORQO4S2TLRL6HMGJQCMJ6PCBZISPKQ, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.859819067 (551h48m43.730902113s) [noise=true , address=145.79.227.49:4367 , piece_id=CITJCIKXK4LYSUV6QRTLIHB7SEH5QOPUM7XR4JDHV54EJ5FLLMDQ, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.861538431 (551h48m43.732621466s) [noise=true , address=145.79.227.33:4904 , piece_id=4B64RHS2HQ6WZYND6MZGU4AZC4LQBXPNKOE2ABNVZQTEBXALQCDA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.863179095 (551h48m43.73426214s) [noise=true , address=77.90.31.19:4874 , piece_id=IXOOASSSDYJS42HWYZKK4PKE4LW47BT4WF5WRRXY7ANPLY632LRA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.864788100 (551h48m43.735871145s) [noise=true , address=77.90.31.16:4749 , piece_id=IVHKFK76TGLMJ7IBVS5ZCCMB35LKHF4FE5HLE7P5QAVGAWGSRHNA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.866240675 (551h48m43.73732371s) [noise=true , address=45.91.116.7:4029 , piece_id=VTKGXA2SWJZHOLLWREHQO66OHQZWHVRK25STFR62PEIJFGD4SG4A, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.867523768 (551h48m43.738606803s) [noise=true , address=45.91.116.6:4998 , piece_id=KCEYZDX2WGV7UCYX4H4BNSR2CWOHGPSEW76LKU2BGC3UDHA5M5SA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:45.143085883 (551h48m45.014168928s) [upload=483809]: Next returned nil segment Dec 28 15:17:45.143099779 (551h48m45.014182824s) [upload=483809]: Waiting for error group managing segments... Dec 28 15:17:45.472957433 (551h48m45.344040468s) [noise=true , address=64.49.13.139:4476 , piece_id=FUSCXA3KPS24G3R5ORJTDOUOU2KMIK5SNBPROHL2CRKMKBRWFN6A, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:45.739300305 (551h48m45.61038334s) [noise=true , address=145.79.227.6:4280 , piece_id=ARMYEAFMP6DPW2JSHWHOYPJ6OUUSUQETJCSSQQXFQX4DDCVZYLYQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:45.743884264 (551h48m45.614967309s) [noise=true , address=145.79.227.21:4508 , piece_id=PGAOV6UBGJPU7M5L3IR2LQU5LSISUJVKKBB7ZRV233F5MBEAWK6A, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:45.936004292 (551h48m45.807087328s) [noise=true , address=77.90.31.15:4727 , piece_id=YJDXXQTMZQRYCEAKLZUFPW6JO7RWHJMJAOFMURYFNLQSJ4LJG2CQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:45.968797306 (551h48m45.839880351s) [noise=true , address=77.90.31.16:4749 , piece_id=IVHKFK76TGLMJ7IBVS5ZCCMB35LKHF4FE5HLE7P5QAVGAWGSRHNA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.007515762 (551h48m45.878598797s) [noise=true , address=77.90.31.11:4580 , piece_id=UUPDJ75SL3HOJREC2SXJGQMUVLLP5WLN2L3LEON6KIKPB24DEKPQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.147397009 (551h48m46.018480054s) [noise=true , address=45.91.116.7:4029 , piece_id=VTKGXA2SWJZHOLLWREHQO66OHQZWHVRK25STFR62PEIJFGD4SG4A, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.460756007 (551h48m46.331839042s) [noise=true , address=193.149.164.143:4694 , piece_id=SDK5DIHWVFOZQIHQWJMZLOTQJEKRDBUTQ7R7ZTKOKF5LGECJRMOA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.662946081 (551h48m46.534029127s) [noise=true , address=145.79.227.43:4167 , piece_id=WEXP7LDX6O7KPXA6XIMRWNTMGNGSTE3XRFODPNMWFKR3D5WXX3LQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.695276604 (551h48m46.566359639s) [noise=true , address=45.91.116.6:4998 , piece_id=KCEYZDX2WGV7UCYX4H4BNSR2CWOHGPSEW76LKU2BGC3UDHA5M5SA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.739405984 (551h48m46.610489029s) [noise=true , address=145.79.227.33:4904 , piece_id=4B64RHS2HQ6WZYND6MZGU4AZC4LQBXPNKOE2ABNVZQTEBXALQCDA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.764293434 (551h48m46.635376479s) [noise=true , address=145.79.227.40:4083 , piece_id=RSMZDEGC4MX5TFRFKI33WNDK42NZDQ4M6UPRQWO4GE6VWGCJ6AFA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.771299524 (551h48m46.64238256s) [noise=true , address=145.79.227.30:4766 , piece_id=YA4HOOMXQRQCPE3VMQ6WEXBM3ZIED4RZLVFHZHRLBQJFQBFP2F7A, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.791263379 (551h48m46.662346414s) [noise=true , address=77.90.31.19:4874 , piece_id=IXOOASSSDYJS42HWYZKK4PKE4LW47BT4WF5WRRXY7ANPLY632LRA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.791282065 (551h48m46.6623651s) [seg_pos={0 0}, upload=483809]: Segment reached optimal threshold of 14 pieces. Dec 28 15:17:46.791451286 (551h48m46.662534321s) [noise=true , address=145.79.227.49:4367 , piece_id=CITJCIKXK4LYSUV6QRTLIHB7SEH5QOPUM7XR4JDHV54EJ5FLLMDQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF Dec 28 15:17:46.791458450 (551h48m46.662541496s) [noise=true , address=145.79.227.49:4367 , piece_id=CITJCIKXK4LYSUV6QRTLIHB7SEH5QOPUM7XR4JDHV54EJ5FLLMDQ, seg_pos={0 0}, upload=483809]: Long tail cancellation due to optimal threshold Dec 28 15:17:46.791491023 (551h48m46.662574058s) [noise=true , address=77.90.31.12:4610 , piece_id=RSI6EFWFTX4JZUBEFTV3VPMJWMWI7VDMUHG6DMK7CSKTH7MDFGXA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi): protocol: expected piece hash; context canceled; EOF Dec 28 15:17:46.791502935 (551h48m46.66258598s) [noise=true , address=77.90.31.12:4610 , piece_id=RSI6EFWFTX4JZUBEFTV3VPMJWMWI7VDMUHG6DMK7CSKTH7MDFGXA, seg_pos={0 0}, upload=483809]: Long tail cancellation due to optimal threshold Dec 28 15:17:46.791556076 (551h48m46.662639111s) [noise=true , address=145.79.227.34:4932 , piece_id=5NGLBPM6NRZIR4NJTA43KIFQQTD6DS6B37DIFCHXGT5YMDGNMXKA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qJfPZErJx4XGBw7NmfCLS7BogRnN29rUibuWkzQDFUNvPBpmm): protocol: expected piece hash; context canceled; EOF Dec 28 15:17:46.791561086 (551h48m46.662644131s) [noise=true , address=145.79.227.34:4932 , piece_id=5NGLBPM6NRZIR4NJTA43KIFQQTD6DS6B37DIFCHXGT5YMDGNMXKA, seg_pos={0 0}, upload=483809]: Long tail cancellation due to optimal threshold Dec 28 15:17:46.791659873 (551h48m46.662742908s) [noise=true , address=64.49.13.169:4716 , piece_id=V3PN2MUBRFNCLCYOVHYLXUORQO4S2TLRL6HMGJQCMJ6PCBZISPKQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:48986->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:48986->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48986->64.49.13.169:4716: use of closed network connection Dec 28 15:17:46.791665625 (551h48m46.66274866s) [noise=true , address=64.49.13.169:4716 , piece_id=V3PN2MUBRFNCLCYOVHYLXUORQO4S2TLRL6HMGJQCMJ6PCBZISPKQ, seg_pos={0 0}, upload=483809]: Long tail cancellation due to optimal threshold Dec 28 15:17:46.791739836 (551h48m46.662822881s) [seg_pos={0 0}, upload=483809]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 15:17:46.791808337 (551h48m46.662891372s) [upload=483809]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 15:17:46.791812865 (551h48m46.6628959s) [upload=483809]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 15:17:47.097154206 (551h48m46.968237251s) [upload=483809]: Done uploading segments... Dec 28 15:17:47.097173253 (551h48m46.968256288s) [upload=483809]: Upload finished. err: <nil> Duration: 43.222996256s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [...X......X....X..........................X....X...X........] duration: 43.59806181s trace: https://jaeger.dev.storj.io/trace/51e2216a4efa8b95 FAIL (2025-12-28 15:25:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 15s) [noise=true , address=145.79.227.40:4083 , piece_id=RSMZDEGC4MX5TFRFKI33WNDK42NZDQ4M6UPRQWO4GE6VWGCJ6AFA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.854006008 (551h48m43.725089043s) [noise=true , address=77.90.31.12:4610 , piece_id=RSI6EFWFTX4JZUBEFTV3VPMJWMWI7VDMUHG6DMK7CSKTH7MDFGXA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.854333892 (551h48m43.725416937s) [noise=true , address=64.49.13.139:4476 , piece_id=FUSCXA3KPS24G3R5ORJTDOUOU2KMIK5SNBPROHL2CRKMKBRWFN6A, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.854753131 (551h48m43.725836176s) [noise=true , address=145.79.227.34:4932 , piece_id=5NGLBPM6NRZIR4NJTA43KIFQQTD6DS6B37DIFCHXGT5YMDGNMXKA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.855102245 (551h48m43.726185381s) [noise=true , address=77.90.31.15:4727 , piece_id=YJDXXQTMZQRYCEAKLZUFPW6JO7RWHJMJAOFMURYFNLQSJ4LJG2CQ, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.855467040 (551h48m43.726550076s) [noise=true , address=145.79.227.6:4280 , piece_id=ARMYEAFMP6DPW2JSHWHOYPJ6OUUSUQETJCSSQQXFQX4DDCVZYLYQ, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.855834780 (551h48m43.726917815s) [noise=true , address=193.149.164.143:4694 , piece_id=SDK5DIHWVFOZQIHQWJMZLOTQJEKRDBUTQ7R7ZTKOKF5LGECJRMOA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.856263167 (551h48m43.727346212s) [noise=true , address=145.79.227.43:4167 , piece_id=WEXP7LDX6O7KPXA6XIMRWNTMGNGSTE3XRFODPNMWFKR3D5WXX3LQ, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.858002919 (551h48m43.729085964s) [noise=true , address=145.79.227.30:4766 , piece_id=YA4HOOMXQRQCPE3VMQ6WEXBM3ZIED4RZLVFHZHRLBQJFQBFP2F7A, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.858063876 (551h48m43.729146911s) [noise=true , address=64.49.13.169:4716 , piece_id=V3PN2MUBRFNCLCYOVHYLXUORQO4S2TLRL6HMGJQCMJ6PCBZISPKQ, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.859819067 (551h48m43.730902113s) [noise=true , address=145.79.227.49:4367 , piece_id=CITJCIKXK4LYSUV6QRTLIHB7SEH5QOPUM7XR4JDHV54EJ5FLLMDQ, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.861538431 (551h48m43.732621466s) [noise=true , address=145.79.227.33:4904 , piece_id=4B64RHS2HQ6WZYND6MZGU4AZC4LQBXPNKOE2ABNVZQTEBXALQCDA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.863179095 (551h48m43.73426214s) [noise=true , address=77.90.31.19:4874 , piece_id=IXOOASSSDYJS42HWYZKK4PKE4LW47BT4WF5WRRXY7ANPLY632LRA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.864788100 (551h48m43.735871145s) [noise=true , address=77.90.31.16:4749 , piece_id=IVHKFK76TGLMJ7IBVS5ZCCMB35LKHF4FE5HLE7P5QAVGAWGSRHNA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.866240675 (551h48m43.73732371s) [noise=true , address=45.91.116.7:4029 , piece_id=VTKGXA2SWJZHOLLWREHQO66OHQZWHVRK25STFR62PEIJFGD4SG4A, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.867523768 (551h48m43.738606803s) [noise=true , address=45.91.116.6:4998 , piece_id=KCEYZDX2WGV7UCYX4H4BNSR2CWOHGPSEW76LKU2BGC3UDHA5M5SA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:45.143085883 (551h48m45.014168928s) [upload=483809]: Next returned nil segment Dec 28 15:17:45.143099779 (551h48m45.014182824s) [upload=483809]: Waiting for error group managing segments... Dec 28 15:17:45.472957433 (551h48m45.344040468s) [noise=true , address=64.49.13.139:4476 , piece_id=FUSCXA3KPS24G3R5ORJTDOUOU2KMIK5SNBPROHL2CRKMKBRWFN6A, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:45.739300305 (551h48m45.61038334s) [noise=true , address=145.79.227.6:4280 , piece_id=ARMYEAFMP6DPW2JSHWHOYPJ6OUUSUQETJCSSQQXFQX4DDCVZYLYQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:45.743884264 (551h48m45.614967309s) [noise=true , address=145.79.227.21:4508 , piece_id=PGAOV6UBGJPU7M5L3IR2LQU5LSISUJVKKBB7ZRV233F5MBEAWK6A, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:45.936004292 (551h48m45.807087328s) [noise=true , address=77.90.31.15:4727 , piece_id=YJDXXQTMZQRYCEAKLZUFPW6JO7RWHJMJAOFMURYFNLQSJ4LJG2CQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:45.968797306 (551h48m45.839880351s) [noise=true , address=77.90.31.16:4749 , piece_id=IVHKFK76TGLMJ7IBVS5ZCCMB35LKHF4FE5HLE7P5QAVGAWGSRHNA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.007515762 (551h48m45.878598797s) [noise=true , address=77.90.31.11:4580 , piece_id=UUPDJ75SL3HOJREC2SXJGQMUVLLP5WLN2L3LEON6KIKPB24DEKPQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.147397009 (551h48m46.018480054s) [noise=true , address=45.91.116.7:4029 , piece_id=VTKGXA2SWJZHOLLWREHQO66OHQZWHVRK25STFR62PEIJFGD4SG4A, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.460756007 (551h48m46.331839042s) [noise=true , address=193.149.164.143:4694 , piece_id=SDK5DIHWVFOZQIHQWJMZLOTQJEKRDBUTQ7R7ZTKOKF5LGECJRMOA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.662946081 (551h48m46.534029127s) [noise=true , address=145.79.227.43:4167 , piece_id=WEXP7LDX6O7KPXA6XIMRWNTMGNGSTE3XRFODPNMWFKR3D5WXX3LQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.695276604 (551h48m46.566359639s) [noise=true , address=45.91.116.6:4998 , piece_id=KCEYZDX2WGV7UCYX4H4BNSR2CWOHGPSEW76LKU2BGC3UDHA5M5SA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.739405984 (551h48m46.610489029s) [noise=true , address=145.79.227.33:4904 , piece_id=4B64RHS2HQ6WZYND6MZGU4AZC4LQBXPNKOE2ABNVZQTEBXALQCDA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.764293434 (551h48m46.635376479s) [noise=true , address=145.79.227.40:4083 , piece_id=RSMZDEGC4MX5TFRFKI33WNDK42NZDQ4M6UPRQWO4GE6VWGCJ6AFA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.771299524 (551h48m46.64238256s) [noise=true , address=145.79.227.30:4766 , piece_id=YA4HOOMXQRQCPE3VMQ6WEXBM3ZIED4RZLVFHZHRLBQJFQBFP2F7A, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.791263379 (551h48m46.662346414s) [noise=true , address=77.90.31.19:4874 , piece_id=IXOOASSSDYJS42HWYZKK4PKE4LW47BT4WF5WRRXY7ANPLY632LRA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.791282065 (551h48m46.6623651s) [seg_pos={0 0}, upload=483809]: Segment reached optimal threshold of 14 pieces. Dec 28 15:17:46.791451286 (551h48m46.662534321s) [noise=true , address=145.79.227.49:4367 , piece_id=CITJCIKXK4LYSUV6QRTLIHB7SEH5QOPUM7XR4JDHV54EJ5FLLMDQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF Dec 28 15:17:46.791458450 (551h48m46.662541496s) [noise=true , address=145.79.227.49:4367 , piece_id=CITJCIKXK4LYSUV6QRTLIHB7SEH5QOPUM7XR4JDHV54EJ5FLLMDQ, seg_pos={0 0}, upload=483809]: Long tail cancellation due to optimal threshold Dec 28 15:17:46.791491023 (551h48m46.662574058s) [noise=true , address=77.90.31.12:4610 , piece_id=RSI6EFWFTX4JZUBEFTV3VPMJWMWI7VDMUHG6DMK7CSKTH7MDFGXA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi): protocol: expected piece hash; context canceled; EOF Dec 28 15:17:46.791502935 (551h48m46.66258598s) [noise=true , address=77.90.31.12:4610 , piece_id=RSI6EFWFTX4JZUBEFTV3VPMJWMWI7VDMUHG6DMK7CSKTH7MDFGXA, seg_pos={0 0}, upload=483809]: Long tail cancellation due to optimal threshold Dec 28 15:17:46.791556076 (551h48m46.662639111s) [noise=true , address=145.79.227.34:4932 , piece_id=5NGLBPM6NRZIR4NJTA43KIFQQTD6DS6B37DIFCHXGT5YMDGNMXKA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qJfPZErJx4XGBw7NmfCLS7BogRnN29rUibuWkzQDFUNvPBpmm): protocol: expected piece hash; context canceled; EOF Dec 28 15:17:46.791561086 (551h48m46.662644131s) [noise=true , address=145.79.227.34:4932 , piece_id=5NGLBPM6NRZIR4NJTA43KIFQQTD6DS6B37DIFCHXGT5YMDGNMXKA, seg_pos={0 0}, upload=483809]: Long tail cancellation due to optimal threshold Dec 28 15:17:46.791659873 (551h48m46.662742908s) [noise=true , address=64.49.13.169:4716 , piece_id=V3PN2MUBRFNCLCYOVHYLXUORQO4S2TLRL6HMGJQCMJ6PCBZISPKQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:48986->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:48986->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48986->64.49.13.169:4716: use of closed network connection Dec 28 15:17:46.791665625 (551h48m46.66274866s) [noise=true , address=64.49.13.169:4716 , piece_id=V3PN2MUBRFNCLCYOVHYLXUORQO4S2TLRL6HMGJQCMJ6PCBZISPKQ, seg_pos={0 0}, upload=483809]: Long tail cancellation due to optimal threshold Dec 28 15:17:46.791739836 (551h48m46.662822881s) [seg_pos={0 0}, upload=483809]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 15:17:46.791808337 (551h48m46.662891372s) [upload=483809]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 15:17:46.791812865 (551h48m46.6628959s) [upload=483809]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 15:17:47.097154206 (551h48m46.968237251s) [upload=483809]: Done uploading segments... Dec 28 15:17:47.097173253 (551h48m46.968256288s) [upload=483809]: Upload finished. err: <nil> Duration: 43.598078843s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 12.352986836s trace: https://jaeger.dev.storj.io/trace/4ce8174d0eaf2b7b FAIL (2025-12-28 15:25:04 UTC) orbiter warning: context deadline exceeded Duration: 12.353014179s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 40 out of 60 is failed (threshold=6) [XXX.X..XXXXX.X.X.XXXXX...X.X.XX.XXX.X.XXXXX.XX.XX.X.XXXXX.X.] duration: 3.523399443s trace: https://jaeger.dev.storj.io/trace/10b88f7e4b3f6704 FAIL (2025-12-28 15:25:03 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.523538959s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [X...X.X.........X.....X......X.X.X.....X......XX.X..XXX.X.XX] duration: 30.00313041s trace: https://jaeger.dev.storj.io/trace/479830a23de6de2e FAIL (2025-12-28 15:25:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, Piece ID: TUO5TA2EOERHHGNASAL3CF2LFWNSUJ5DOUOFN25UTJYAFN6TBR7Q) context deadline exceeded; piecestore: (Node ID: 1aRnwUHwyB11FpSDzQWDikCaEu6YCnvKwy9VRFYq8sbm7K6jgQ, Piece ID: DNSUMNQFWPHMIYTJEVYTOBNEFZOGDF6OGF57SHELSD5W7K5P45BA) context deadline exceeded ======LOGS====== 28 15:24:31.745302663 (551h55m31.616385698s) [upload=483905]: Starting upload Dec 28 15:24:31.745385191 (551h55m31.616468236s) [upload=483905]: Done starting upload Dec 28 15:24:31.745421751 (551h55m31.616504796s) [upload=483905]: Uploading segments... Dec 28 15:24:31.745602915 (551h55m31.616685961s) [upload=483905]: Got next segment. Inline: false Dec 28 15:24:31.745690011 (551h55m31.616773056s) [upload=483905]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 15:24:31.745700411 (551h55m31.616783446s) [upload=483905]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 15:24:31.996545582 (551h55m31.867628617s) [seg_pos={0 0}, upload=483905]: Begin upload segment... Dec 28 15:24:31.996604164 (551h55m31.867687209s) [seg_pos={0 0}, upload=483905]: Done begin upload segment. Dec 28 15:24:31.996646644 (551h55m31.867729679s) [noise=true , address=145.79.227.37:4566 , piece_id=OH35DIHAR6CWSNVJ3VCBZDZ2XLEXCI626NACGFSVLW7EDID3RZDA, seg_pos={0 0}, upload=483905]: Uploading piece... Dec 28 15:24:31.996654990 (551h55m31.867738025s) [noise=true , address=45.91.116.13:4155 , piece_id=2KXM65GO346ZTPHKZSVYY6PCJ6NXPJ6JA4V5E7TVWYNUFAHSRVEQ, seg_pos={0 0}, upload=483905]: Uploading piece... Dec 28 15:24:31.997214806 (551h55m31.868297851s) [noise=true , address=145.79.227.13:4802 , piece_id=3ADM4MSYHR3KWSZGQDQQRU3W2G6HB3Y53VEPBBRKE5WDYTXCNPPA, seg_pos={0 0}, upload=483905]: Uploading piece... Dec 28 15:24:31.997661056 (551h55m31.868744091s) [noise=true , address=145.79.227.52:4445 , piece_id=F6XYJUKZJZZEPHGMOKIA3KU7DNKE5BFV3ECEDX6UDDA74KWD5LQA, seg_pos={0 0}, upload=483905]: Uploading piece... Dec 28 15:24:31.997845196 (551h55m31.868928231s) [noise=true , address=77.90.31.11:4555 , piece_id=DNSUMNQFWPHMIYTJEVYTOBNEFZOGDF6OGF57SHELSD5W7K5P45BA, seg_pos={0 0}, upload=483905]: Uploading piece... Dec 28 15:24:31.998252171 (551h55m31.869335207s) [noise=true , address=145.79.227.6:4258 , piece_id=DEG6OQAK4WMFUSUEMR2Y4SUOPII5PWX7LPBYZO2NJZXGRDN7VZFA, seg_pos={0 0}, upload=483905]: Uploading piece... Dec 28 15:24:31.999032647 (551h55m31.870115682s) [noise=true , address=145.79.227.35:4481 , piece_id=NUBA7KB6C5O6XVXQGGYUNS343IXIU4ZMI7XKDVKVDKQNIUK67HWQ, seg_pos={0 0}, upload=483905]: Uploading piece... Dec 28 15:24:31.999120285 (551h55m31.87020333s) [noise=true , address=145.79.227.39:4014 , piece_id=2N3BK5OUAVZWR2SLJZIBB3FODZYJKB5RXBWNJY5THODKVEDJRG5Q, seg_pos={0 0}, upload=483905]: Uploading piece... Dec 28 15:24:32.000022473 (551h55m31.871105528s) [noise=true , address=145.79.227.19:4079 , piece_id=TKSNPV3YS64OGRA6PV7HQSJTPURN3F2BKS723YBBZMOLB5XHZIKA, seg_pos={0 0}, upload=483905]: Uploading piece... Dec 28 15:24:32.002082174 (551h55m31.873165219s) [noise=true , address=145.79.227.36:4519 , piece_id=VD3KC3DNOXXL2P65V55WF56GDVIJIBUD2JEDLYQ4XYNFEYFDQNWQ, seg_pos={0 0}, upload=483905]: Uploading piece... Dec 28 15:24:32.002728185 (551h55m31.87381122s) [noise=true , address=145.79.227.51:4410 , piece_id=BXWBBO3GPF2W4JM6CYAZGIU6OTSK4EHR6CCTGBASRUDCL6HO4B2A, seg_pos={0 0}, upload=483905]: Uploading piece... Dec 28 15:24:32.003124269 (551h55m31.874207314s) [noise=true , address=77.90.31.21:4878 , piece_id=4JJ2ZRFSFGRV7LGJHBWXKEIQIQHASUUWCH4YKZOW5BMKFA4VERSA, seg_pos={0 0}, upload=483905]: Uploading piece... Dec 28 15:24:32.003711657 (551h55m31.874794713s) [noise=true , address=77.90.31.18:4805 , piece_id=TUO5TA2EOERHHGNASAL3CF2LFWNSUJ5DOUOFN25UTJYAFN6TBR7Q, seg_pos={0 0}, upload=483905]: Uploading piece... Dec 28 15:24:32.003973215 (551h55m31.87505625s) [noise=true , address=145.79.227.11:4729 , piece_id=CBOV4FDOHKRKTV2EQJQ32WHX3XB24CPZ3NNHLQGM56H5UU6O6FQA, seg_pos={0 0}, upload=483905]: Uploading piece... Dec 28 15:24:32.004384979 (551h55m31.875468014s) [noise=true , address=145.79.227.31:4798 , piece_id=5OGCVZDH46CU42NBSAX7CB45EAWBJJ2QRPMCSY5EUTNDAVGNP6CA, seg_pos={0 0}, upload=483905]: Uploading piece... Dec 28 15:24:32.004472597 (551h55m31.875555632s) [noise=true , address=145.79.227.45:4230 , piece_id=NOBAPNBCOMPFKEZPBOOHLCPOJF6RQISC3WA5MFJYG2Z2NVM42A4A, seg_pos={0 0}, upload=483905]: Uploading piece... Dec 28 15:24:33.718603700 (551h55m33.589686745s) [upload=483905]: Next returned nil segment Dec 28 15:24:33.718621222 (551h55m33.589704268s) [upload=483905]: Waiting for error group managing segments... Dec 28 15:24:34.005201960 (551h55m33.876284995s) [noise=true , address=145.79.227.19:4079 , piece_id=TKSNPV3YS64OGRA6PV7HQSJTPURN3F2BKS723YBBZMOLB5XHZIKA, seg_pos={0 0}, upload=483905]: Done uploading piece. err: <nil> Dec 28 15:24:34.006150686 (551h55m33.877233721s) [noise=true , address=145.79.227.6:4258 , piece_id=DEG6OQAK4WMFUSUEMR2Y4SUOPII5PWX7LPBYZO2NJZXGRDN7VZFA, seg_pos={0 0}, upload=483905]: Done uploading piece. err: <nil> Dec 28 15:24:34.007109342 (551h55m33.878192377s) [noise=true , address=145.79.227.13:4802 , piece_id=3ADM4MSYHR3KWSZGQDQQRU3W2G6HB3Y53VEPBBRKE5WDYTXCNPPA, seg_pos={0 0}, upload=483905]: Done uploading piece. err: <nil> Dec 28 15:24:34.023006242 (551h55m33.894089287s) [noise=true , address=145.79.227.36:4519 , piece_id=VD3KC3DNOXXL2P65V55WF56GDVIJIBUD2JEDLYQ4XYNFEYFDQNWQ, seg_pos={0 0}, upload=483905]: Done uploading piece. err: <nil> Dec 28 15:24:34.026062011 (551h55m33.897145056s) [noise=true , address=145.79.227.52:4445 , piece_id=F6XYJUKZJZZEPHGMOKIA3KU7DNKE5BFV3ECEDX6UDDA74KWD5LQA, seg_pos={0 0}, upload=483905]: Done uploading piece. err: <nil> Dec 28 15:24:34.026771031 (551h55m33.897854066s) [noise=true , address=145.79.227.31:4798 , piece_id=5OGCVZDH46CU42NBSAX7CB45EAWBJJ2QRPMCSY5EUTNDAVGNP6CA, seg_pos={0 0}, upload=483905]: Done uploading piece. err: <nil> Dec 28 15:24:34.027002271 (551h55m33.898085307s) [noise=true , address=145.79.227.11:4729 , piece_id=CBOV4FDOHKRKTV2EQJQ32WHX3XB24CPZ3NNHLQGM56H5UU6O6FQA, seg_pos={0 0}, upload=483905]: Done uploading piece. err: <nil> Dec 28 15:24:34.027305990 (551h55m33.898389035s) [noise=true , address=145.79.227.37:4566 , piece_id=OH35DIHAR6CWSNVJ3VCBZDZ2XLEXCI626NACGFSVLW7EDID3RZDA, seg_pos={0 0}, upload=483905]: Done uploading piece. err: <nil> Dec 28 15:24:34.028476448 (551h55m33.899559483s) [noise=true , address=145.79.227.51:4410 , piece_id=BXWBBO3GPF2W4JM6CYAZGIU6OTSK4EHR6CCTGBASRUDCL6HO4B2A, seg_pos={0 0}, upload=483905]: Done uploading piece. err: <nil> Dec 28 15:24:34.029970814 (551h55m33.901053859s) [noise=true , address=145.79.227.45:4230 , piece_id=NOBAPNBCOMPFKEZPBOOHLCPOJF6RQISC3WA5MFJYG2Z2NVM42A4A, seg_pos={0 0}, upload=483905]: Done uploading piece. err: <nil> Dec 28 15:24:34.032421892 (551h55m33.903504937s) [noise=true , address=145.79.227.39:4014 , piece_id=2N3BK5OUAVZWR2SLJZIBB3FODZYJKB5RXBWNJY5THODKVEDJRG5Q, seg_pos={0 0}, upload=483905]: Done uploading piece. err: <nil> Dec 28 15:24:34.528252686 (551h55m34.399335722s) [noise=true , address=77.90.31.18:4805 , piece_id=TUO5TA2EOERHHGNASAL3CF2LFWNSUJ5DOUOFN25UTJYAFN6TBR7Q, seg_pos={0 0}, upload=483905]: Done uploading piece. err: <nil> Dec 28 15:24:34.872423013 (551h55m34.743506048s) [noise=true , address=77.90.31.11:4555 , piece_id=DNSUMNQFWPHMIYTJEVYTOBNEFZOGDF6OGF57SHELSD5W7K5P45BA, seg_pos={0 0}, upload=483905]: Done uploading piece. err: <nil> Dec 28 15:24:34.872443082 (551h55m34.743526117s) [seg_pos={0 0}, upload=483905]: Segment reached optimal threshold of 13 pieces. Dec 28 15:24:34.872626781 (551h55m34.743709817s) [noise=true , address=77.90.31.21:4878 , piece_id=4JJ2ZRFSFGRV7LGJHBWXKEIQIQHASUUWCH4YKZOW5BMKFA4VERSA, seg_pos={0 0}, upload=483905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: write tcp 188.166.225.170:40656->77.90.31.21:4878: use of closed network connection; write tcp 188.166.225.170:40656->77.90.31.21:4878: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40656->77.90.31.21:4878: use of closed network connection Dec 28 15:24:34.872634576 (551h55m34.743717611s) [noise=true , address=77.90.31.21:4878 , piece_id=4JJ2ZRFSFGRV7LGJHBWXKEIQIQHASUUWCH4YKZOW5BMKFA4VERSA, seg_pos={0 0}, upload=483905]: Long tail cancellation due to optimal threshold Dec 28 15:24:34.872794431 (551h55m34.743877466s) [noise=true , address=145.79.227.35:4481 , piece_id=NUBA7KB6C5O6XVXQGGYUNS343IXIU4ZMI7XKDVKVDKQNIUK67HWQ, seg_pos={0 0}, upload=483905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp): protocol: expected piece hash; context canceled; EOF Dec 28 15:24:34.872802075 (551h55m34.74388511s) [noise=true , address=145.79.227.35:4481 , piece_id=NUBA7KB6C5O6XVXQGGYUNS343IXIU4ZMI7XKDVKVDKQNIUK67HWQ, seg_pos={0 0}, upload=483905]: Long tail cancellation due to optimal threshold Dec 28 15:24:34.872827814 (551h55m34.74391085s) [noise=true , address=45.91.116.13:4155 , piece_id=2KXM65GO346ZTPHKZSVYY6PCJ6NXPJ6JA4V5E7TVWYNUFAHSRVEQ, seg_pos={0 0}, upload=483905]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf): protocol: expected piece hash; context canceled; EOF Dec 28 15:24:34.872838815 (551h55m34.74392185s) [noise=true , address=45.91.116.13:4155 , piece_id=2KXM65GO346ZTPHKZSVYY6PCJ6NXPJ6JA4V5E7TVWYNUFAHSRVEQ, seg_pos={0 0}, upload=483905]: Long tail cancellation due to optimal threshold Dec 28 15:24:34.872855998 (551h55m34.743939033s) [seg_pos={0 0}, upload=483905]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 15:24:34.872940298 (551h55m34.744023333s) [upload=483905]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 15:24:34.872945378 (551h55m34.744028413s) [upload=483905]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 15:24:35.162244914 (551h55m35.033327959s) [upload=483905]: Done uploading segments... Dec 28 15:24:35.162263268 (551h55m35.033346303s) [upload=483905]: Upload finished. err: <nil> Duration: 30.003154025s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 11.384271545s trace: https://jaeger.dev.storj.io/trace/4ce8174d0eaf2b7b FAIL (2025-12-28 15:24:46 UTC) orbiter warning: context deadline exceeded Duration: 11.384316632s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [....X......X....X..........................X....X...X.......] duration: 44.469868491s trace: https://jaeger.dev.storj.io/trace/51e2216a4efa8b95 FAIL (2025-12-28 15:24:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 15s) [noise=true , address=145.79.227.40:4083 , piece_id=RSMZDEGC4MX5TFRFKI33WNDK42NZDQ4M6UPRQWO4GE6VWGCJ6AFA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.854006008 (551h48m43.725089043s) [noise=true , address=77.90.31.12:4610 , piece_id=RSI6EFWFTX4JZUBEFTV3VPMJWMWI7VDMUHG6DMK7CSKTH7MDFGXA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.854333892 (551h48m43.725416937s) [noise=true , address=64.49.13.139:4476 , piece_id=FUSCXA3KPS24G3R5ORJTDOUOU2KMIK5SNBPROHL2CRKMKBRWFN6A, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.854753131 (551h48m43.725836176s) [noise=true , address=145.79.227.34:4932 , piece_id=5NGLBPM6NRZIR4NJTA43KIFQQTD6DS6B37DIFCHXGT5YMDGNMXKA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.855102245 (551h48m43.726185381s) [noise=true , address=77.90.31.15:4727 , piece_id=YJDXXQTMZQRYCEAKLZUFPW6JO7RWHJMJAOFMURYFNLQSJ4LJG2CQ, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.855467040 (551h48m43.726550076s) [noise=true , address=145.79.227.6:4280 , piece_id=ARMYEAFMP6DPW2JSHWHOYPJ6OUUSUQETJCSSQQXFQX4DDCVZYLYQ, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.855834780 (551h48m43.726917815s) [noise=true , address=193.149.164.143:4694 , piece_id=SDK5DIHWVFOZQIHQWJMZLOTQJEKRDBUTQ7R7ZTKOKF5LGECJRMOA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.856263167 (551h48m43.727346212s) [noise=true , address=145.79.227.43:4167 , piece_id=WEXP7LDX6O7KPXA6XIMRWNTMGNGSTE3XRFODPNMWFKR3D5WXX3LQ, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.858002919 (551h48m43.729085964s) [noise=true , address=145.79.227.30:4766 , piece_id=YA4HOOMXQRQCPE3VMQ6WEXBM3ZIED4RZLVFHZHRLBQJFQBFP2F7A, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.858063876 (551h48m43.729146911s) [noise=true , address=64.49.13.169:4716 , piece_id=V3PN2MUBRFNCLCYOVHYLXUORQO4S2TLRL6HMGJQCMJ6PCBZISPKQ, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.859819067 (551h48m43.730902113s) [noise=true , address=145.79.227.49:4367 , piece_id=CITJCIKXK4LYSUV6QRTLIHB7SEH5QOPUM7XR4JDHV54EJ5FLLMDQ, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.861538431 (551h48m43.732621466s) [noise=true , address=145.79.227.33:4904 , piece_id=4B64RHS2HQ6WZYND6MZGU4AZC4LQBXPNKOE2ABNVZQTEBXALQCDA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.863179095 (551h48m43.73426214s) [noise=true , address=77.90.31.19:4874 , piece_id=IXOOASSSDYJS42HWYZKK4PKE4LW47BT4WF5WRRXY7ANPLY632LRA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.864788100 (551h48m43.735871145s) [noise=true , address=77.90.31.16:4749 , piece_id=IVHKFK76TGLMJ7IBVS5ZCCMB35LKHF4FE5HLE7P5QAVGAWGSRHNA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.866240675 (551h48m43.73732371s) [noise=true , address=45.91.116.7:4029 , piece_id=VTKGXA2SWJZHOLLWREHQO66OHQZWHVRK25STFR62PEIJFGD4SG4A, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.867523768 (551h48m43.738606803s) [noise=true , address=45.91.116.6:4998 , piece_id=KCEYZDX2WGV7UCYX4H4BNSR2CWOHGPSEW76LKU2BGC3UDHA5M5SA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:45.143085883 (551h48m45.014168928s) [upload=483809]: Next returned nil segment Dec 28 15:17:45.143099779 (551h48m45.014182824s) [upload=483809]: Waiting for error group managing segments... Dec 28 15:17:45.472957433 (551h48m45.344040468s) [noise=true , address=64.49.13.139:4476 , piece_id=FUSCXA3KPS24G3R5ORJTDOUOU2KMIK5SNBPROHL2CRKMKBRWFN6A, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:45.739300305 (551h48m45.61038334s) [noise=true , address=145.79.227.6:4280 , piece_id=ARMYEAFMP6DPW2JSHWHOYPJ6OUUSUQETJCSSQQXFQX4DDCVZYLYQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:45.743884264 (551h48m45.614967309s) [noise=true , address=145.79.227.21:4508 , piece_id=PGAOV6UBGJPU7M5L3IR2LQU5LSISUJVKKBB7ZRV233F5MBEAWK6A, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:45.936004292 (551h48m45.807087328s) [noise=true , address=77.90.31.15:4727 , piece_id=YJDXXQTMZQRYCEAKLZUFPW6JO7RWHJMJAOFMURYFNLQSJ4LJG2CQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:45.968797306 (551h48m45.839880351s) [noise=true , address=77.90.31.16:4749 , piece_id=IVHKFK76TGLMJ7IBVS5ZCCMB35LKHF4FE5HLE7P5QAVGAWGSRHNA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.007515762 (551h48m45.878598797s) [noise=true , address=77.90.31.11:4580 , piece_id=UUPDJ75SL3HOJREC2SXJGQMUVLLP5WLN2L3LEON6KIKPB24DEKPQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.147397009 (551h48m46.018480054s) [noise=true , address=45.91.116.7:4029 , piece_id=VTKGXA2SWJZHOLLWREHQO66OHQZWHVRK25STFR62PEIJFGD4SG4A, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.460756007 (551h48m46.331839042s) [noise=true , address=193.149.164.143:4694 , piece_id=SDK5DIHWVFOZQIHQWJMZLOTQJEKRDBUTQ7R7ZTKOKF5LGECJRMOA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.662946081 (551h48m46.534029127s) [noise=true , address=145.79.227.43:4167 , piece_id=WEXP7LDX6O7KPXA6XIMRWNTMGNGSTE3XRFODPNMWFKR3D5WXX3LQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.695276604 (551h48m46.566359639s) [noise=true , address=45.91.116.6:4998 , piece_id=KCEYZDX2WGV7UCYX4H4BNSR2CWOHGPSEW76LKU2BGC3UDHA5M5SA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.739405984 (551h48m46.610489029s) [noise=true , address=145.79.227.33:4904 , piece_id=4B64RHS2HQ6WZYND6MZGU4AZC4LQBXPNKOE2ABNVZQTEBXALQCDA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.764293434 (551h48m46.635376479s) [noise=true , address=145.79.227.40:4083 , piece_id=RSMZDEGC4MX5TFRFKI33WNDK42NZDQ4M6UPRQWO4GE6VWGCJ6AFA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.771299524 (551h48m46.64238256s) [noise=true , address=145.79.227.30:4766 , piece_id=YA4HOOMXQRQCPE3VMQ6WEXBM3ZIED4RZLVFHZHRLBQJFQBFP2F7A, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.791263379 (551h48m46.662346414s) [noise=true , address=77.90.31.19:4874 , piece_id=IXOOASSSDYJS42HWYZKK4PKE4LW47BT4WF5WRRXY7ANPLY632LRA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.791282065 (551h48m46.6623651s) [seg_pos={0 0}, upload=483809]: Segment reached optimal threshold of 14 pieces. Dec 28 15:17:46.791451286 (551h48m46.662534321s) [noise=true , address=145.79.227.49:4367 , piece_id=CITJCIKXK4LYSUV6QRTLIHB7SEH5QOPUM7XR4JDHV54EJ5FLLMDQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF Dec 28 15:17:46.791458450 (551h48m46.662541496s) [noise=true , address=145.79.227.49:4367 , piece_id=CITJCIKXK4LYSUV6QRTLIHB7SEH5QOPUM7XR4JDHV54EJ5FLLMDQ, seg_pos={0 0}, upload=483809]: Long tail cancellation due to optimal threshold Dec 28 15:17:46.791491023 (551h48m46.662574058s) [noise=true , address=77.90.31.12:4610 , piece_id=RSI6EFWFTX4JZUBEFTV3VPMJWMWI7VDMUHG6DMK7CSKTH7MDFGXA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi): protocol: expected piece hash; context canceled; EOF Dec 28 15:17:46.791502935 (551h48m46.66258598s) [noise=true , address=77.90.31.12:4610 , piece_id=RSI6EFWFTX4JZUBEFTV3VPMJWMWI7VDMUHG6DMK7CSKTH7MDFGXA, seg_pos={0 0}, upload=483809]: Long tail cancellation due to optimal threshold Dec 28 15:17:46.791556076 (551h48m46.662639111s) [noise=true , address=145.79.227.34:4932 , piece_id=5NGLBPM6NRZIR4NJTA43KIFQQTD6DS6B37DIFCHXGT5YMDGNMXKA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qJfPZErJx4XGBw7NmfCLS7BogRnN29rUibuWkzQDFUNvPBpmm): protocol: expected piece hash; context canceled; EOF Dec 28 15:17:46.791561086 (551h48m46.662644131s) [noise=true , address=145.79.227.34:4932 , piece_id=5NGLBPM6NRZIR4NJTA43KIFQQTD6DS6B37DIFCHXGT5YMDGNMXKA, seg_pos={0 0}, upload=483809]: Long tail cancellation due to optimal threshold Dec 28 15:17:46.791659873 (551h48m46.662742908s) [noise=true , address=64.49.13.169:4716 , piece_id=V3PN2MUBRFNCLCYOVHYLXUORQO4S2TLRL6HMGJQCMJ6PCBZISPKQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:48986->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:48986->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48986->64.49.13.169:4716: use of closed network connection Dec 28 15:17:46.791665625 (551h48m46.66274866s) [noise=true , address=64.49.13.169:4716 , piece_id=V3PN2MUBRFNCLCYOVHYLXUORQO4S2TLRL6HMGJQCMJ6PCBZISPKQ, seg_pos={0 0}, upload=483809]: Long tail cancellation due to optimal threshold Dec 28 15:17:46.791739836 (551h48m46.662822881s) [seg_pos={0 0}, upload=483809]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 15:17:46.791808337 (551h48m46.662891372s) [upload=483809]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 15:17:46.791812865 (551h48m46.6628959s) [upload=483809]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 15:17:47.097154206 (551h48m46.968237251s) [upload=483809]: Done uploading segments... Dec 28 15:17:47.097173253 (551h48m46.968256288s) [upload=483809]: Upload finished. err: <nil> Duration: 44.469884451s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 41 out of 60 is failed (threshold=6) [XXXX.X..XXXXX.X.X.XXXXX...X.X.XX.XXX.X.XXXXX.XX.XX.X.XXXXX.X] duration: 5.001607668s trace: https://jaeger.dev.storj.io/trace/10b88f7e4b3f6704 FAIL (2025-12-28 15:24:21 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00161968s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 13.525055821s trace: https://jaeger.dev.storj.io/trace/4ce8174d0eaf2b7b FAIL (2025-12-28 15:24:17 UTC) orbiter warning: context deadline exceeded Duration: 13.525085607s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.X...X.X.........X.....X......X.X.X.....X......XX.X..XXX.X.X] duration: 30.003299069s trace: https://jaeger.dev.storj.io/trace/a45ca5ebc588e9c FAIL (2025-12-28 15:24:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== c 28 15:23:32.036294954 (551h54m31.907377989s) [upload=483891]: Starting upload Dec 28 15:23:32.036384435 (551h54m31.90746747s) [upload=483891]: Done starting upload Dec 28 15:23:32.036416636 (551h54m31.907499671s) [upload=483891]: Uploading segments... Dec 28 15:23:32.036575508 (551h54m31.907658543s) [upload=483891]: Got next segment. Inline: false Dec 28 15:23:32.036608451 (551h54m31.907691496s) [upload=483891]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 15:23:32.036612979 (551h54m31.907696014s) [upload=483891]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 15:23:32.289834637 (551h54m32.160917702s) [seg_pos={0 0}, upload=483891]: Begin upload segment... Dec 28 15:23:32.289902356 (551h54m32.160985411s) [seg_pos={0 0}, upload=483891]: Done begin upload segment. Dec 28 15:23:32.289955397 (551h54m32.161038432s) [noise=true , address=145.79.227.13:4804 , piece_id=FJIMRY2IHN3MN2QQO7RQP7PA2MABU7EPGFKQUFXVPAV27GCX7WSA, seg_pos={0 0}, upload=483891]: Uploading piece... Dec 28 15:23:32.289989913 (551h54m32.161072958s) [noise=true , address=77.90.31.19:4842 , piece_id=SU7BWHQEOUE4M4SY4ZL3RI7RV2IZHODEIYK4KMJ6PY2BDMNJ6S5Q, seg_pos={0 0}, upload=483891]: Uploading piece... Dec 28 15:23:32.290577833 (551h54m32.161660878s) [noise=true , address=77.90.31.12:4589 , piece_id=WL6J4SVEVI43D7BAE2SQJ2F4D2BSREV6KR6YYSE3B325LKGGCR5A, seg_pos={0 0}, upload=483891]: Uploading piece... Dec 28 15:23:32.290977543 (551h54m32.162060578s) [noise=true , address=145.79.227.23:4546 , piece_id=COAV44BGVQWXL4DOS3J45RS7EBXAEWCWTDPDHCVO6ZFZR5V6H4TQ, seg_pos={0 0}, upload=483891]: Uploading piece... Dec 28 15:23:32.291365292 (551h54m32.162448327s) [noise=true , address=45.91.116.12:4225 , piece_id=XTV2V3QR2NQ2VRET7CK357RSQP64L3RGL6GGAGD67ZTX7IWGA3QQ, seg_pos={0 0}, upload=483891]: Uploading piece... Dec 28 15:23:32.291765184 (551h54m32.162848229s) [noise=true , address=45.91.116.9:4263 , piece_id=USASA33A5ICXTZBVRZNEPLB5G2MJ42EKNATBPLSQOJ7T46R6HZ5A, seg_pos={0 0}, upload=483891]: Uploading piece... Dec 28 15:23:32.292175475 (551h54m32.16325852s) [noise=true , address=45.91.116.11:4120 , piece_id=6ZLX6OZL3JJBIDXO32P76QZVYN4ZAXROKQR2ZXXK7BXJ2WO3VKYA, seg_pos={0 0}, upload=483891]: Uploading piece... Dec 28 15:23:32.292666570 (551h54m32.163749615s) [noise=true , address=45.91.116.8:4048 , piece_id=2FXMXMZ6FBJYXM3L7BSNBNTM75HQZEYCEC4CAN4BPVYNGQTH6MSA, seg_pos={0 0}, upload=483891]: Uploading piece... Dec 28 15:23:32.293573146 (551h54m32.164656181s) [noise=true , address=77.90.31.11:4553 , piece_id=GPL3NYUPSKRHKCV64SR3KRMCWELFE37GZMLMPVRKF72ROGMSXWAQ, seg_pos={0 0}, upload=483891]: Uploading piece... Dec 28 15:23:32.294198788 (551h54m32.165281823s) [noise=true , address=45.91.116.10:4082 , piece_id=HKXXHOPUSPXQEMIBN2VAKWCXJ6N734EUIU57AWPEQUIIMPADEFXQ, seg_pos={0 0}, upload=483891]: Uploading piece... Dec 28 15:23:32.294792438 (551h54m32.165875473s) [noise=true , address=145.79.227.38:4977 , piece_id=67S7AU3VV76PUKRT2XHUTSHUU5ULHNPUWITRSAQBFRQGOB4CL73Q, seg_pos={0 0}, upload=483891]: Uploading piece... Dec 28 15:23:32.295814374 (551h54m32.166897409s) [noise=true , address=145.79.227.11:4731 , piece_id=ZYWSBJGJELWW7WNITCSQ7E3ARDFVHQW2RHDZXKUIXDRRRWL7P7BA, seg_pos={0 0}, upload=483891]: Uploading piece... Dec 28 15:23:32.297169404 (551h54m32.168252439s) [noise=true , address=145.79.227.47:4267 , piece_id=MMDDJDA2R3LM3UKCF3JPPET7ARVKPKY5YVMKXO3ICY2MBZICV7RA, seg_pos={0 0}, upload=483891]: Uploading piece... Dec 28 15:23:32.299151769 (551h54m32.170234834s) [noise=true , address=145.79.227.33:4869 , piece_id=BBLR5B6GWJN4WSP2NYLAW7SVYU5IQZNJGBDEZ5S25LFG66NBEWWQ, seg_pos={0 0}, upload=483891]: Uploading piece... Dec 28 15:23:32.300313761 (551h54m32.171396796s) [noise=true , address=145.79.227.2:4115 , piece_id=AY3GXE73CYBGNJOFEYS6M3DADTNQ2NM2XWY6EX7CTD37S4LWBEWA, seg_pos={0 0}, upload=483891]: Uploading piece... Dec 28 15:23:32.301408996 (551h54m32.172492041s) [noise=true , address=45.91.116.14:4189 , piece_id=M53OADOTSNZON4HAAEBTLKPPKLWTSXUWCGDHZALZGRKFU6FJRPLQ, seg_pos={0 0}, upload=483891]: Uploading piece... Dec 28 15:23:34.014722362 (551h54m33.885805408s) [upload=483891]: Next returned nil segment Dec 28 15:23:34.014744846 (551h54m33.885827891s) [upload=483891]: Waiting for error group managing segments... Dec 28 15:23:34.300415730 (551h54m34.171498775s) [noise=true , address=145.79.227.2:4115 , piece_id=AY3GXE73CYBGNJOFEYS6M3DADTNQ2NM2XWY6EX7CTD37S4LWBEWA, seg_pos={0 0}, upload=483891]: Done uploading piece. err: <nil> Dec 28 15:23:34.303478672 (551h54m34.174561717s) [noise=true , address=145.79.227.13:4804 , piece_id=FJIMRY2IHN3MN2QQO7RQP7PA2MABU7EPGFKQUFXVPAV27GCX7WSA, seg_pos={0 0}, upload=483891]: Done uploading piece. err: <nil> Dec 28 15:23:34.316459360 (551h54m34.187542405s) [noise=true , address=145.79.227.47:4267 , piece_id=MMDDJDA2R3LM3UKCF3JPPET7ARVKPKY5YVMKXO3ICY2MBZICV7RA, seg_pos={0 0}, upload=483891]: Done uploading piece. err: <nil> Dec 28 15:23:34.320188020 (551h54m34.191271085s) [noise=true , address=145.79.227.23:4546 , piece_id=COAV44BGVQWXL4DOS3J45RS7EBXAEWCWTDPDHCVO6ZFZR5V6H4TQ, seg_pos={0 0}, upload=483891]: Done uploading piece. err: <nil> Dec 28 15:23:34.323676693 (551h54m34.194759728s) [noise=true , address=145.79.227.38:4977 , piece_id=67S7AU3VV76PUKRT2XHUTSHUU5ULHNPUWITRSAQBFRQGOB4CL73Q, seg_pos={0 0}, upload=483891]: Done uploading piece. err: <nil> Dec 28 15:23:34.326047236 (551h54m34.197130271s) [noise=true , address=145.79.227.11:4731 , piece_id=ZYWSBJGJELWW7WNITCSQ7E3ARDFVHQW2RHDZXKUIXDRRRWL7P7BA, seg_pos={0 0}, upload=483891]: Done uploading piece. err: <nil> Dec 28 15:23:34.489991424 (551h54m34.361074469s) [noise=true , address=45.91.116.11:4120 , piece_id=6ZLX6OZL3JJBIDXO32P76QZVYN4ZAXROKQR2ZXXK7BXJ2WO3VKYA, seg_pos={0 0}, upload=483891]: Done uploading piece. err: <nil> Dec 28 15:23:34.526606325 (551h54m34.39768936s) [noise=true , address=45.91.116.9:4263 , piece_id=USASA33A5ICXTZBVRZNEPLB5G2MJ42EKNATBPLSQOJ7T46R6HZ5A, seg_pos={0 0}, upload=483891]: Done uploading piece. err: <nil> Dec 28 15:23:34.590115056 (551h54m34.461198091s) [noise=true , address=77.90.31.19:4842 , piece_id=SU7BWHQEOUE4M4SY4ZL3RI7RV2IZHODEIYK4KMJ6PY2BDMNJ6S5Q, seg_pos={0 0}, upload=483891]: Done uploading piece. err: <nil> Dec 28 15:23:34.655633735 (551h54m34.52671677s) [noise=true , address=77.90.31.11:4553 , piece_id=GPL3NYUPSKRHKCV64SR3KRMCWELFE37GZMLMPVRKF72ROGMSXWAQ, seg_pos={0 0}, upload=483891]: Done uploading piece. err: <nil> Dec 28 15:23:34.864571758 (551h54m34.735654793s) [noise=true , address=45.91.116.14:4189 , piece_id=M53OADOTSNZON4HAAEBTLKPPKLWTSXUWCGDHZALZGRKFU6FJRPLQ, seg_pos={0 0}, upload=483891]: Done uploading piece. err: <nil> Dec 28 15:23:34.993626399 (551h54m34.864709434s) [noise=true , address=45.91.116.8:4048 , piece_id=2FXMXMZ6FBJYXM3L7BSNBNTM75HQZEYCEC4CAN4BPVYNGQTH6MSA, seg_pos={0 0}, upload=483891]: Done uploading piece. err: <nil> Dec 28 15:23:35.319531688 (551h54m35.190614733s) [noise=true , address=77.90.31.12:4589 , piece_id=WL6J4SVEVI43D7BAE2SQJ2F4D2BSREV6KR6YYSE3B325LKGGCR5A, seg_pos={0 0}, upload=483891]: Done uploading piece. err: <nil> Dec 28 15:23:35.319551184 (551h54m35.190634219s) [seg_pos={0 0}, upload=483891]: Segment reached optimal threshold of 13 pieces. Dec 28 15:23:35.319713604 (551h54m35.190796649s) [noise=true , address=45.91.116.10:4082 , piece_id=HKXXHOPUSPXQEMIBN2VAKWCXJ6N734EUIU57AWPEQUIIMPADEFXQ, seg_pos={0 0}, upload=483891]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 28 15:23:35.319719675 (551h54m35.19080271s) [noise=true , address=45.91.116.10:4082 , piece_id=HKXXHOPUSPXQEMIBN2VAKWCXJ6N734EUIU57AWPEQUIIMPADEFXQ, seg_pos={0 0}, upload=483891]: Long tail cancellation due to optimal threshold Dec 28 15:23:35.319852619 (551h54m35.190935654s) [noise=true , address=145.79.227.33:4869 , piece_id=BBLR5B6GWJN4WSP2NYLAW7SVYU5IQZNJGBDEZ5S25LFG66NBEWWQ, seg_pos={0 0}, upload=483891]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:18CQxjq65r6xVTmNRurSQPzKxj4BoR6cJ8Uq29sTrdh6uXm5sq): protocol: expected piece hash; context canceled; EOF Dec 28 15:23:35.319866004 (551h54m35.190949049s) [noise=true , address=145.79.227.33:4869 , piece_id=BBLR5B6GWJN4WSP2NYLAW7SVYU5IQZNJGBDEZ5S25LFG66NBEWWQ, seg_pos={0 0}, upload=483891]: Long tail cancellation due to optimal threshold Dec 28 15:23:35.319997554 (551h54m35.1910806s) [noise=true , address=45.91.116.12:4225 , piece_id=XTV2V3QR2NQ2VRET7CK357RSQP64L3RGL6GGAGD67ZTX7IWGA3QQ, seg_pos={0 0}, upload=483891]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AXaXCJFwDBoz9beNXsHYyfdgRkcX684iB6msjy9d5FkmwCWhy): protocol: write tcp 188.166.225.170:39570->45.91.116.12:4225: use of closed network connection; write tcp 188.166.225.170:39570->45.91.116.12:4225: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39570->45.91.116.12:4225: use of closed network connection Dec 28 15:23:35.320002794 (551h54m35.191085839s) [noise=true , address=45.91.116.12:4225 , piece_id=XTV2V3QR2NQ2VRET7CK357RSQP64L3RGL6GGAGD67ZTX7IWGA3QQ, seg_pos={0 0}, upload=483891]: Long tail cancellation due to optimal threshold Dec 28 15:23:35.320017894 (551h54m35.191100929s) [seg_pos={0 0}, upload=483891]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 15:23:35.320106331 (551h54m35.191189377s) [upload=483891]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 15:23:35.320123695 (551h54m35.19120675s) [upload=483891]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 15:23:35.614325124 (551h54m35.485408169s) [upload=483891]: Done uploading segments... Dec 28 15:23:35.614343278 (551h54m35.485426313s) [upload=483891]: Upload finished. err: <nil> Duration: 30.003325649s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 22.865442735s trace: https://jaeger.dev.storj.io/trace/4ce8174d0eaf2b7b FAIL (2025-12-28 15:23:49 UTC) orbiter warning: context deadline exceeded Duration: 22.865457863s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....X......X....X..........................X....X...X......] duration: 39.962161927s trace: https://jaeger.dev.storj.io/trace/51e2216a4efa8b95 FAIL (2025-12-28 15:23:43 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 15s) [noise=true , address=145.79.227.40:4083 , piece_id=RSMZDEGC4MX5TFRFKI33WNDK42NZDQ4M6UPRQWO4GE6VWGCJ6AFA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.854006008 (551h48m43.725089043s) [noise=true , address=77.90.31.12:4610 , piece_id=RSI6EFWFTX4JZUBEFTV3VPMJWMWI7VDMUHG6DMK7CSKTH7MDFGXA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.854333892 (551h48m43.725416937s) [noise=true , address=64.49.13.139:4476 , piece_id=FUSCXA3KPS24G3R5ORJTDOUOU2KMIK5SNBPROHL2CRKMKBRWFN6A, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.854753131 (551h48m43.725836176s) [noise=true , address=145.79.227.34:4932 , piece_id=5NGLBPM6NRZIR4NJTA43KIFQQTD6DS6B37DIFCHXGT5YMDGNMXKA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.855102245 (551h48m43.726185381s) [noise=true , address=77.90.31.15:4727 , piece_id=YJDXXQTMZQRYCEAKLZUFPW6JO7RWHJMJAOFMURYFNLQSJ4LJG2CQ, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.855467040 (551h48m43.726550076s) [noise=true , address=145.79.227.6:4280 , piece_id=ARMYEAFMP6DPW2JSHWHOYPJ6OUUSUQETJCSSQQXFQX4DDCVZYLYQ, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.855834780 (551h48m43.726917815s) [noise=true , address=193.149.164.143:4694 , piece_id=SDK5DIHWVFOZQIHQWJMZLOTQJEKRDBUTQ7R7ZTKOKF5LGECJRMOA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.856263167 (551h48m43.727346212s) [noise=true , address=145.79.227.43:4167 , piece_id=WEXP7LDX6O7KPXA6XIMRWNTMGNGSTE3XRFODPNMWFKR3D5WXX3LQ, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.858002919 (551h48m43.729085964s) [noise=true , address=145.79.227.30:4766 , piece_id=YA4HOOMXQRQCPE3VMQ6WEXBM3ZIED4RZLVFHZHRLBQJFQBFP2F7A, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.858063876 (551h48m43.729146911s) [noise=true , address=64.49.13.169:4716 , piece_id=V3PN2MUBRFNCLCYOVHYLXUORQO4S2TLRL6HMGJQCMJ6PCBZISPKQ, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.859819067 (551h48m43.730902113s) [noise=true , address=145.79.227.49:4367 , piece_id=CITJCIKXK4LYSUV6QRTLIHB7SEH5QOPUM7XR4JDHV54EJ5FLLMDQ, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.861538431 (551h48m43.732621466s) [noise=true , address=145.79.227.33:4904 , piece_id=4B64RHS2HQ6WZYND6MZGU4AZC4LQBXPNKOE2ABNVZQTEBXALQCDA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.863179095 (551h48m43.73426214s) [noise=true , address=77.90.31.19:4874 , piece_id=IXOOASSSDYJS42HWYZKK4PKE4LW47BT4WF5WRRXY7ANPLY632LRA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.864788100 (551h48m43.735871145s) [noise=true , address=77.90.31.16:4749 , piece_id=IVHKFK76TGLMJ7IBVS5ZCCMB35LKHF4FE5HLE7P5QAVGAWGSRHNA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.866240675 (551h48m43.73732371s) [noise=true , address=45.91.116.7:4029 , piece_id=VTKGXA2SWJZHOLLWREHQO66OHQZWHVRK25STFR62PEIJFGD4SG4A, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.867523768 (551h48m43.738606803s) [noise=true , address=45.91.116.6:4998 , piece_id=KCEYZDX2WGV7UCYX4H4BNSR2CWOHGPSEW76LKU2BGC3UDHA5M5SA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:45.143085883 (551h48m45.014168928s) [upload=483809]: Next returned nil segment Dec 28 15:17:45.143099779 (551h48m45.014182824s) [upload=483809]: Waiting for error group managing segments... Dec 28 15:17:45.472957433 (551h48m45.344040468s) [noise=true , address=64.49.13.139:4476 , piece_id=FUSCXA3KPS24G3R5ORJTDOUOU2KMIK5SNBPROHL2CRKMKBRWFN6A, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:45.739300305 (551h48m45.61038334s) [noise=true , address=145.79.227.6:4280 , piece_id=ARMYEAFMP6DPW2JSHWHOYPJ6OUUSUQETJCSSQQXFQX4DDCVZYLYQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:45.743884264 (551h48m45.614967309s) [noise=true , address=145.79.227.21:4508 , piece_id=PGAOV6UBGJPU7M5L3IR2LQU5LSISUJVKKBB7ZRV233F5MBEAWK6A, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:45.936004292 (551h48m45.807087328s) [noise=true , address=77.90.31.15:4727 , piece_id=YJDXXQTMZQRYCEAKLZUFPW6JO7RWHJMJAOFMURYFNLQSJ4LJG2CQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:45.968797306 (551h48m45.839880351s) [noise=true , address=77.90.31.16:4749 , piece_id=IVHKFK76TGLMJ7IBVS5ZCCMB35LKHF4FE5HLE7P5QAVGAWGSRHNA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.007515762 (551h48m45.878598797s) [noise=true , address=77.90.31.11:4580 , piece_id=UUPDJ75SL3HOJREC2SXJGQMUVLLP5WLN2L3LEON6KIKPB24DEKPQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.147397009 (551h48m46.018480054s) [noise=true , address=45.91.116.7:4029 , piece_id=VTKGXA2SWJZHOLLWREHQO66OHQZWHVRK25STFR62PEIJFGD4SG4A, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.460756007 (551h48m46.331839042s) [noise=true , address=193.149.164.143:4694 , piece_id=SDK5DIHWVFOZQIHQWJMZLOTQJEKRDBUTQ7R7ZTKOKF5LGECJRMOA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.662946081 (551h48m46.534029127s) [noise=true , address=145.79.227.43:4167 , piece_id=WEXP7LDX6O7KPXA6XIMRWNTMGNGSTE3XRFODPNMWFKR3D5WXX3LQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.695276604 (551h48m46.566359639s) [noise=true , address=45.91.116.6:4998 , piece_id=KCEYZDX2WGV7UCYX4H4BNSR2CWOHGPSEW76LKU2BGC3UDHA5M5SA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.739405984 (551h48m46.610489029s) [noise=true , address=145.79.227.33:4904 , piece_id=4B64RHS2HQ6WZYND6MZGU4AZC4LQBXPNKOE2ABNVZQTEBXALQCDA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.764293434 (551h48m46.635376479s) [noise=true , address=145.79.227.40:4083 , piece_id=RSMZDEGC4MX5TFRFKI33WNDK42NZDQ4M6UPRQWO4GE6VWGCJ6AFA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.771299524 (551h48m46.64238256s) [noise=true , address=145.79.227.30:4766 , piece_id=YA4HOOMXQRQCPE3VMQ6WEXBM3ZIED4RZLVFHZHRLBQJFQBFP2F7A, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.791263379 (551h48m46.662346414s) [noise=true , address=77.90.31.19:4874 , piece_id=IXOOASSSDYJS42HWYZKK4PKE4LW47BT4WF5WRRXY7ANPLY632LRA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.791282065 (551h48m46.6623651s) [seg_pos={0 0}, upload=483809]: Segment reached optimal threshold of 14 pieces. Dec 28 15:17:46.791451286 (551h48m46.662534321s) [noise=true , address=145.79.227.49:4367 , piece_id=CITJCIKXK4LYSUV6QRTLIHB7SEH5QOPUM7XR4JDHV54EJ5FLLMDQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF Dec 28 15:17:46.791458450 (551h48m46.662541496s) [noise=true , address=145.79.227.49:4367 , piece_id=CITJCIKXK4LYSUV6QRTLIHB7SEH5QOPUM7XR4JDHV54EJ5FLLMDQ, seg_pos={0 0}, upload=483809]: Long tail cancellation due to optimal threshold Dec 28 15:17:46.791491023 (551h48m46.662574058s) [noise=true , address=77.90.31.12:4610 , piece_id=RSI6EFWFTX4JZUBEFTV3VPMJWMWI7VDMUHG6DMK7CSKTH7MDFGXA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi): protocol: expected piece hash; context canceled; EOF Dec 28 15:17:46.791502935 (551h48m46.66258598s) [noise=true , address=77.90.31.12:4610 , piece_id=RSI6EFWFTX4JZUBEFTV3VPMJWMWI7VDMUHG6DMK7CSKTH7MDFGXA, seg_pos={0 0}, upload=483809]: Long tail cancellation due to optimal threshold Dec 28 15:17:46.791556076 (551h48m46.662639111s) [noise=true , address=145.79.227.34:4932 , piece_id=5NGLBPM6NRZIR4NJTA43KIFQQTD6DS6B37DIFCHXGT5YMDGNMXKA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qJfPZErJx4XGBw7NmfCLS7BogRnN29rUibuWkzQDFUNvPBpmm): protocol: expected piece hash; context canceled; EOF Dec 28 15:17:46.791561086 (551h48m46.662644131s) [noise=true , address=145.79.227.34:4932 , piece_id=5NGLBPM6NRZIR4NJTA43KIFQQTD6DS6B37DIFCHXGT5YMDGNMXKA, seg_pos={0 0}, upload=483809]: Long tail cancellation due to optimal threshold Dec 28 15:17:46.791659873 (551h48m46.662742908s) [noise=true , address=64.49.13.169:4716 , piece_id=V3PN2MUBRFNCLCYOVHYLXUORQO4S2TLRL6HMGJQCMJ6PCBZISPKQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:48986->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:48986->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48986->64.49.13.169:4716: use of closed network connection Dec 28 15:17:46.791665625 (551h48m46.66274866s) [noise=true , address=64.49.13.169:4716 , piece_id=V3PN2MUBRFNCLCYOVHYLXUORQO4S2TLRL6HMGJQCMJ6PCBZISPKQ, seg_pos={0 0}, upload=483809]: Long tail cancellation due to optimal threshold Dec 28 15:17:46.791739836 (551h48m46.662822881s) [seg_pos={0 0}, upload=483809]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 15:17:46.791808337 (551h48m46.662891372s) [upload=483809]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 15:17:46.791812865 (551h48m46.6628959s) [upload=483809]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 15:17:47.097154206 (551h48m46.968237251s) [upload=483809]: Done uploading segments... Dec 28 15:17:47.097173253 (551h48m46.968256288s) [upload=483809]: Upload finished. err: <nil> Duration: 39.962191101s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 41 out of 60 is failed (threshold=6) [XXXXX.X..XXXXX.X.X.XXXXX...X.X.XX.XXX.X.XXXXX.XX.XX.X.XXXXX.] duration: 4.839388341s trace: https://jaeger.dev.storj.io/trace/3df2a936138b1b9 FAIL (2025-12-28 15:23:24 UTC) orbiter warning: context deadline exceeded Duration: 4.839433607s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [..X...X.X.........X.....X......X.X.X.....X......XX.X..XXX.X.] duration: 29.670457515s trace: https://jaeger.dev.storj.io/trace/210080892bd48307 FAIL (2025-12-28 15:22:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 15:21:55.690573314 (551h52m55.56165636s) [upload=483869]: Starting upload Dec 28 15:21:55.690634019 (551h52m55.561717064s) [upload=483869]: Done starting upload Dec 28 15:21:55.690705666 (551h52m55.561788711s) [upload=483869]: Uploading segments... Dec 28 15:21:55.690914573 (551h52m55.561997609s) [upload=483869]: Got next segment. Inline: false Dec 28 15:21:55.690946444 (551h52m55.562029489s) [upload=483869]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 15:21:55.690953247 (551h52m55.562036292s) [upload=483869]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 15:21:55.957013671 (551h52m55.828096706s) [seg_pos={0 0}, upload=483869]: Begin upload segment... Dec 28 15:21:55.957095196 (551h52m55.828178231s) [seg_pos={0 0}, upload=483869]: Done begin upload segment. Dec 28 15:21:55.958131419 (551h52m55.829214474s) [noise=true , address=145.79.227.4:4186 , piece_id=DUZI4UWTRCCHICNHAHN3BWJ3OOOM65UIYSLEYMV2J3SYD55S4PJQ, seg_pos={0 0}, upload=483869]: Uploading piece... Dec 28 15:21:55.959605496 (551h52m55.830688531s) [noise=true , address=145.79.227.11:4731 , piece_id=Q7GOF3OCVY7BC7MOJNKE7UJHJJVU57DTNUFJICMTTV7NDWIJG4XQ, seg_pos={0 0}, upload=483869]: Uploading piece... Dec 28 15:21:55.960168058 (551h52m55.831251093s) [noise=true , address=145.79.227.39:4015 , piece_id=VGXABVT3FVXSQQ7DHQWYSUE7S7UCAAP3TWERSTCZYGKNX5A64CWA, seg_pos={0 0}, upload=483869]: Uploading piece... Dec 28 15:21:55.962088605 (551h52m55.83317166s) [noise=true , address=45.91.116.6:4974 , piece_id=Q27YPVZKNCIYZRZKABJL7PM5JUYVTZPMADCN5N5ZJPR6AVJBLKFQ, seg_pos={0 0}, upload=483869]: Uploading piece... Dec 28 15:21:55.962692725 (551h52m55.83377576s) [noise=true , address=145.79.227.48:4302 , piece_id=CFN6LD43EF6IGY65CZU3HBYRTGNX46OKEU2XHG5AUUABMYYLZNZQ, seg_pos={0 0}, upload=483869]: Uploading piece... Dec 28 15:21:55.963247651 (551h52m55.834330686s) [noise=true , address=145.79.227.45:4231 , piece_id=V5Q76OWTPEPBSN6G25EDN6U7JK2EZPIFYU5GC5476AAEKADWO3OA, seg_pos={0 0}, upload=483869]: Uploading piece... Dec 28 15:21:55.963766640 (551h52m55.834849685s) [noise=true , address=45.91.116.5:4298 , piece_id=U3PHTR3AA2ZE2YJYFGZYKHQQMHEOWZFCGQO2PYBBK4IPX66SIA4Q, seg_pos={0 0}, upload=483869]: Uploading piece... Dec 28 15:21:55.964253447 (551h52m55.835336492s) [noise=true , address=145.79.227.36:4518 , piece_id=VQE5JSTUS5A4HWPSCS3DU2GXCZN5GZIGXGH2SJP7GEXNNIFSXKIQ, seg_pos={0 0}, upload=483869]: Uploading piece... Dec 28 15:21:55.964721898 (551h52m55.835804933s) [noise=true , address=145.79.227.46:4602 , piece_id=ZFXQTJOV5THQBEWRUVFNRF4ASX5V4LAZECS3LG2MSSHGIHR77STA, seg_pos={0 0}, upload=483869]: Uploading piece... Dec 28 15:21:55.965446218 (551h52m55.836529253s) [noise=true , address=77.90.31.13:4629 , piece_id=X3WSEQJXSZLPEN7LR4UCHDIBDWKOD6USQ7TNRYDI2YO4SC7PWCKQ, seg_pos={0 0}, upload=483869]: Uploading piece... Dec 28 15:21:55.968439207 (551h52m55.839522252s) [noise=true , address=45.91.116.7:4011 , piece_id=KZGXWS3H7RWVEIGZVNMBF3GWX2R7XA5HCZ6ZCHPHZRBQ4YWBQP7Q, seg_pos={0 0}, upload=483869]: Uploading piece... Dec 28 15:21:55.968848957 (551h52m55.839932002s) [noise=true , address=77.90.31.21:4878 , piece_id=S6EUHFVVYX6FXNSM24MYMTGVO4D5X3QVZZN2TCUYODMFTB4AWJAQ, seg_pos={0 0}, upload=483869]: Uploading piece... Dec 28 15:21:55.969454680 (551h52m55.840537725s) [noise=true , address=45.91.116.13:4153 , piece_id=IUUC2ALVV2APBGNPNFIUVWNFC6PAPGHOSHENGZX4VRUTCPEF6T3Q, seg_pos={0 0}, upload=483869]: Uploading piece... Dec 28 15:21:55.969854442 (551h52m55.840937477s) [noise=true , address=145.79.227.50:4373 , piece_id=ZHFPYVPMSFHWE56EZ6HYH35QMBU6FRCOH5PNPMRM7NYCMZIH6CHA, seg_pos={0 0}, upload=483869]: Uploading piece... Dec 28 15:21:55.978253365 (551h52m55.84933641s) [noise=true , address=45.91.116.11:4119 , piece_id=AOEN4PLL3Y6OOC6B4UDHSXSKE4AAZFENDW4MWJ4N77BCMAMBQTHQ, seg_pos={0 0}, upload=483869]: Uploading piece... Dec 28 15:21:55.985268051 (551h52m55.856351107s) [noise=true , address=145.79.227.41:4087 , piece_id=YN2MX7LEVJVR4JMPXFGMJGOL2DMJUEKI33LFNNBT2A7BN7LTLILA, seg_pos={0 0}, upload=483869]: Uploading piece... Dec 28 15:21:57.695500922 (551h52m57.566583967s) [upload=483869]: Next returned nil segment Dec 28 15:21:57.695519928 (551h52m57.566602993s) [upload=483869]: Waiting for error group managing segments... Dec 28 15:21:57.976465435 (551h52m57.84754848s) [noise=true , address=145.79.227.36:4518 , piece_id=VQE5JSTUS5A4HWPSCS3DU2GXCZN5GZIGXGH2SJP7GEXNNIFSXKIQ, seg_pos={0 0}, upload=483869]: Done uploading piece. err: <nil> Dec 28 15:21:57.980147786 (551h52m57.851230832s) [noise=true , address=145.79.227.50:4373 , piece_id=ZHFPYVPMSFHWE56EZ6HYH35QMBU6FRCOH5PNPMRM7NYCMZIH6CHA, seg_pos={0 0}, upload=483869]: Done uploading piece. err: <nil> Dec 28 15:21:57.980606651 (551h52m57.851689686s) [noise=true , address=145.79.227.4:4186 , piece_id=DUZI4UWTRCCHICNHAHN3BWJ3OOOM65UIYSLEYMV2J3SYD55S4PJQ, seg_pos={0 0}, upload=483869]: Done uploading piece. err: <nil> Dec 28 15:21:57.980669520 (551h52m57.851752555s) [noise=true , address=145.79.227.46:4602 , piece_id=ZFXQTJOV5THQBEWRUVFNRF4ASX5V4LAZECS3LG2MSSHGIHR77STA, seg_pos={0 0}, upload=483869]: Done uploading piece. err: <nil> Dec 28 15:21:57.981590334 (551h52m57.852673379s) [noise=true , address=145.79.227.48:4302 , piece_id=CFN6LD43EF6IGY65CZU3HBYRTGNX46OKEU2XHG5AUUABMYYLZNZQ, seg_pos={0 0}, upload=483869]: Done uploading piece. err: <nil> Dec 28 15:21:57.985324264 (551h52m57.856407309s) [noise=true , address=145.79.227.11:4731 , piece_id=Q7GOF3OCVY7BC7MOJNKE7UJHJJVU57DTNUFJICMTTV7NDWIJG4XQ, seg_pos={0 0}, upload=483869]: Done uploading piece. err: <nil> Dec 28 15:21:57.999049198 (551h52m57.870132233s) [noise=true , address=145.79.227.41:4087 , piece_id=YN2MX7LEVJVR4JMPXFGMJGOL2DMJUEKI33LFNNBT2A7BN7LTLILA, seg_pos={0 0}, upload=483869]: Done uploading piece. err: <nil> Dec 28 15:21:58.000320720 (551h52m57.871403775s) [noise=true , address=145.79.227.45:4231 , piece_id=V5Q76OWTPEPBSN6G25EDN6U7JK2EZPIFYU5GC5476AAEKADWO3OA, seg_pos={0 0}, upload=483869]: Done uploading piece. err: <nil> Dec 28 15:21:58.001534331 (551h52m57.872617386s) [noise=true , address=145.79.227.39:4015 , piece_id=VGXABVT3FVXSQQ7DHQWYSUE7S7UCAAP3TWERSTCZYGKNX5A64CWA, seg_pos={0 0}, upload=483869]: Done uploading piece. err: <nil> Dec 28 15:21:58.260468705 (551h52m58.13155175s) [noise=true , address=77.90.31.13:4629 , piece_id=X3WSEQJXSZLPEN7LR4UCHDIBDWKOD6USQ7TNRYDI2YO4SC7PWCKQ, seg_pos={0 0}, upload=483869]: Done uploading piece. err: <nil> Dec 28 15:21:58.348164904 (551h52m58.219247949s) [noise=true , address=45.91.116.6:4974 , piece_id=Q27YPVZKNCIYZRZKABJL7PM5JUYVTZPMADCN5N5ZJPR6AVJBLKFQ, seg_pos={0 0}, upload=483869]: Done uploading piece. err: <nil> Dec 28 15:21:58.356225171 (551h52m58.227308206s) [noise=true , address=45.91.116.11:4119 , piece_id=AOEN4PLL3Y6OOC6B4UDHSXSKE4AAZFENDW4MWJ4N77BCMAMBQTHQ, seg_pos={0 0}, upload=483869]: Done uploading piece. err: <nil> Dec 28 15:21:58.381959954 (551h52m58.253042989s) [noise=true , address=77.90.31.21:4878 , piece_id=S6EUHFVVYX6FXNSM24MYMTGVO4D5X3QVZZN2TCUYODMFTB4AWJAQ, seg_pos={0 0}, upload=483869]: Done uploading piece. err: <nil> Dec 28 15:21:58.381977728 (551h52m58.253060763s) [seg_pos={0 0}, upload=483869]: Segment reached optimal threshold of 13 pieces. Dec 28 15:21:58.382133495 (551h52m58.25321654s) [noise=true , address=45.91.116.7:4011 , piece_id=KZGXWS3H7RWVEIGZVNMBF3GWX2R7XA5HCZ6ZCHPHZRBQ4YWBQP7Q, seg_pos={0 0}, upload=483869]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 28 15:21:58.382139386 (551h52m58.253222421s) [noise=true , address=45.91.116.7:4011 , piece_id=KZGXWS3H7RWVEIGZVNMBF3GWX2R7XA5HCZ6ZCHPHZRBQ4YWBQP7Q, seg_pos={0 0}, upload=483869]: Long tail cancellation due to optimal threshold Dec 28 15:21:58.382172429 (551h52m58.253255474s) [noise=true , address=45.91.116.5:4298 , piece_id=U3PHTR3AA2ZE2YJYFGZYKHQQMHEOWZFCGQO2PYBBK4IPX66SIA4Q, seg_pos={0 0}, upload=483869]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:39010->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:39010->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39010->45.91.116.5:4298: use of closed network connection Dec 28 15:21:58.382180163 (551h52m58.253263198s) [noise=true , address=45.91.116.5:4298 , piece_id=U3PHTR3AA2ZE2YJYFGZYKHQQMHEOWZFCGQO2PYBBK4IPX66SIA4Q, seg_pos={0 0}, upload=483869]: Long tail cancellation due to optimal threshold Dec 28 15:21:58.382266228 (551h52m58.253349263s) [noise=true , address=45.91.116.13:4153 , piece_id=IUUC2ALVV2APBGNPNFIUVWNFC6PAPGHOSHENGZX4VRUTCPEF6T3Q, seg_pos={0 0}, upload=483869]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 28 15:21:58.382273782 (551h52m58.253356837s) [noise=true , address=45.91.116.13:4153 , piece_id=IUUC2ALVV2APBGNPNFIUVWNFC6PAPGHOSHENGZX4VRUTCPEF6T3Q, seg_pos={0 0}, upload=483869]: Long tail cancellation due to optimal threshold Dec 28 15:21:58.382290523 (551h52m58.253373558s) [seg_pos={0 0}, upload=483869]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 15:21:58.382350488 (551h52m58.253433523s) [upload=483869]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 15:21:58.382356359 (551h52m58.253439404s) [upload=483869]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 15:21:58.726319273 (551h52m58.597402318s) [upload=483869]: Done uploading segments... Dec 28 15:21:58.726342005 (551h52m58.59742504s) [upload=483869]: Upload finished. err: <nil> Duration: 29.67047532s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [......X......X....X..........................X....X...X.....] duration: 52.213197214s trace: https://jaeger.dev.storj.io/trace/51e2216a4efa8b95 FAIL (2025-12-28 15:22:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 15s) [noise=true , address=145.79.227.40:4083 , piece_id=RSMZDEGC4MX5TFRFKI33WNDK42NZDQ4M6UPRQWO4GE6VWGCJ6AFA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.854006008 (551h48m43.725089043s) [noise=true , address=77.90.31.12:4610 , piece_id=RSI6EFWFTX4JZUBEFTV3VPMJWMWI7VDMUHG6DMK7CSKTH7MDFGXA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.854333892 (551h48m43.725416937s) [noise=true , address=64.49.13.139:4476 , piece_id=FUSCXA3KPS24G3R5ORJTDOUOU2KMIK5SNBPROHL2CRKMKBRWFN6A, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.854753131 (551h48m43.725836176s) [noise=true , address=145.79.227.34:4932 , piece_id=5NGLBPM6NRZIR4NJTA43KIFQQTD6DS6B37DIFCHXGT5YMDGNMXKA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.855102245 (551h48m43.726185381s) [noise=true , address=77.90.31.15:4727 , piece_id=YJDXXQTMZQRYCEAKLZUFPW6JO7RWHJMJAOFMURYFNLQSJ4LJG2CQ, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.855467040 (551h48m43.726550076s) [noise=true , address=145.79.227.6:4280 , piece_id=ARMYEAFMP6DPW2JSHWHOYPJ6OUUSUQETJCSSQQXFQX4DDCVZYLYQ, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.855834780 (551h48m43.726917815s) [noise=true , address=193.149.164.143:4694 , piece_id=SDK5DIHWVFOZQIHQWJMZLOTQJEKRDBUTQ7R7ZTKOKF5LGECJRMOA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.856263167 (551h48m43.727346212s) [noise=true , address=145.79.227.43:4167 , piece_id=WEXP7LDX6O7KPXA6XIMRWNTMGNGSTE3XRFODPNMWFKR3D5WXX3LQ, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.858002919 (551h48m43.729085964s) [noise=true , address=145.79.227.30:4766 , piece_id=YA4HOOMXQRQCPE3VMQ6WEXBM3ZIED4RZLVFHZHRLBQJFQBFP2F7A, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.858063876 (551h48m43.729146911s) [noise=true , address=64.49.13.169:4716 , piece_id=V3PN2MUBRFNCLCYOVHYLXUORQO4S2TLRL6HMGJQCMJ6PCBZISPKQ, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.859819067 (551h48m43.730902113s) [noise=true , address=145.79.227.49:4367 , piece_id=CITJCIKXK4LYSUV6QRTLIHB7SEH5QOPUM7XR4JDHV54EJ5FLLMDQ, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.861538431 (551h48m43.732621466s) [noise=true , address=145.79.227.33:4904 , piece_id=4B64RHS2HQ6WZYND6MZGU4AZC4LQBXPNKOE2ABNVZQTEBXALQCDA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.863179095 (551h48m43.73426214s) [noise=true , address=77.90.31.19:4874 , piece_id=IXOOASSSDYJS42HWYZKK4PKE4LW47BT4WF5WRRXY7ANPLY632LRA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.864788100 (551h48m43.735871145s) [noise=true , address=77.90.31.16:4749 , piece_id=IVHKFK76TGLMJ7IBVS5ZCCMB35LKHF4FE5HLE7P5QAVGAWGSRHNA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.866240675 (551h48m43.73732371s) [noise=true , address=45.91.116.7:4029 , piece_id=VTKGXA2SWJZHOLLWREHQO66OHQZWHVRK25STFR62PEIJFGD4SG4A, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.867523768 (551h48m43.738606803s) [noise=true , address=45.91.116.6:4998 , piece_id=KCEYZDX2WGV7UCYX4H4BNSR2CWOHGPSEW76LKU2BGC3UDHA5M5SA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:45.143085883 (551h48m45.014168928s) [upload=483809]: Next returned nil segment Dec 28 15:17:45.143099779 (551h48m45.014182824s) [upload=483809]: Waiting for error group managing segments... Dec 28 15:17:45.472957433 (551h48m45.344040468s) [noise=true , address=64.49.13.139:4476 , piece_id=FUSCXA3KPS24G3R5ORJTDOUOU2KMIK5SNBPROHL2CRKMKBRWFN6A, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:45.739300305 (551h48m45.61038334s) [noise=true , address=145.79.227.6:4280 , piece_id=ARMYEAFMP6DPW2JSHWHOYPJ6OUUSUQETJCSSQQXFQX4DDCVZYLYQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:45.743884264 (551h48m45.614967309s) [noise=true , address=145.79.227.21:4508 , piece_id=PGAOV6UBGJPU7M5L3IR2LQU5LSISUJVKKBB7ZRV233F5MBEAWK6A, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:45.936004292 (551h48m45.807087328s) [noise=true , address=77.90.31.15:4727 , piece_id=YJDXXQTMZQRYCEAKLZUFPW6JO7RWHJMJAOFMURYFNLQSJ4LJG2CQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:45.968797306 (551h48m45.839880351s) [noise=true , address=77.90.31.16:4749 , piece_id=IVHKFK76TGLMJ7IBVS5ZCCMB35LKHF4FE5HLE7P5QAVGAWGSRHNA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.007515762 (551h48m45.878598797s) [noise=true , address=77.90.31.11:4580 , piece_id=UUPDJ75SL3HOJREC2SXJGQMUVLLP5WLN2L3LEON6KIKPB24DEKPQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.147397009 (551h48m46.018480054s) [noise=true , address=45.91.116.7:4029 , piece_id=VTKGXA2SWJZHOLLWREHQO66OHQZWHVRK25STFR62PEIJFGD4SG4A, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.460756007 (551h48m46.331839042s) [noise=true , address=193.149.164.143:4694 , piece_id=SDK5DIHWVFOZQIHQWJMZLOTQJEKRDBUTQ7R7ZTKOKF5LGECJRMOA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.662946081 (551h48m46.534029127s) [noise=true , address=145.79.227.43:4167 , piece_id=WEXP7LDX6O7KPXA6XIMRWNTMGNGSTE3XRFODPNMWFKR3D5WXX3LQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.695276604 (551h48m46.566359639s) [noise=true , address=45.91.116.6:4998 , piece_id=KCEYZDX2WGV7UCYX4H4BNSR2CWOHGPSEW76LKU2BGC3UDHA5M5SA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.739405984 (551h48m46.610489029s) [noise=true , address=145.79.227.33:4904 , piece_id=4B64RHS2HQ6WZYND6MZGU4AZC4LQBXPNKOE2ABNVZQTEBXALQCDA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.764293434 (551h48m46.635376479s) [noise=true , address=145.79.227.40:4083 , piece_id=RSMZDEGC4MX5TFRFKI33WNDK42NZDQ4M6UPRQWO4GE6VWGCJ6AFA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.771299524 (551h48m46.64238256s) [noise=true , address=145.79.227.30:4766 , piece_id=YA4HOOMXQRQCPE3VMQ6WEXBM3ZIED4RZLVFHZHRLBQJFQBFP2F7A, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.791263379 (551h48m46.662346414s) [noise=true , address=77.90.31.19:4874 , piece_id=IXOOASSSDYJS42HWYZKK4PKE4LW47BT4WF5WRRXY7ANPLY632LRA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.791282065 (551h48m46.6623651s) [seg_pos={0 0}, upload=483809]: Segment reached optimal threshold of 14 pieces. Dec 28 15:17:46.791451286 (551h48m46.662534321s) [noise=true , address=145.79.227.49:4367 , piece_id=CITJCIKXK4LYSUV6QRTLIHB7SEH5QOPUM7XR4JDHV54EJ5FLLMDQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF Dec 28 15:17:46.791458450 (551h48m46.662541496s) [noise=true , address=145.79.227.49:4367 , piece_id=CITJCIKXK4LYSUV6QRTLIHB7SEH5QOPUM7XR4JDHV54EJ5FLLMDQ, seg_pos={0 0}, upload=483809]: Long tail cancellation due to optimal threshold Dec 28 15:17:46.791491023 (551h48m46.662574058s) [noise=true , address=77.90.31.12:4610 , piece_id=RSI6EFWFTX4JZUBEFTV3VPMJWMWI7VDMUHG6DMK7CSKTH7MDFGXA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi): protocol: expected piece hash; context canceled; EOF Dec 28 15:17:46.791502935 (551h48m46.66258598s) [noise=true , address=77.90.31.12:4610 , piece_id=RSI6EFWFTX4JZUBEFTV3VPMJWMWI7VDMUHG6DMK7CSKTH7MDFGXA, seg_pos={0 0}, upload=483809]: Long tail cancellation due to optimal threshold Dec 28 15:17:46.791556076 (551h48m46.662639111s) [noise=true , address=145.79.227.34:4932 , piece_id=5NGLBPM6NRZIR4NJTA43KIFQQTD6DS6B37DIFCHXGT5YMDGNMXKA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qJfPZErJx4XGBw7NmfCLS7BogRnN29rUibuWkzQDFUNvPBpmm): protocol: expected piece hash; context canceled; EOF Dec 28 15:17:46.791561086 (551h48m46.662644131s) [noise=true , address=145.79.227.34:4932 , piece_id=5NGLBPM6NRZIR4NJTA43KIFQQTD6DS6B37DIFCHXGT5YMDGNMXKA, seg_pos={0 0}, upload=483809]: Long tail cancellation due to optimal threshold Dec 28 15:17:46.791659873 (551h48m46.662742908s) [noise=true , address=64.49.13.169:4716 , piece_id=V3PN2MUBRFNCLCYOVHYLXUORQO4S2TLRL6HMGJQCMJ6PCBZISPKQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:48986->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:48986->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48986->64.49.13.169:4716: use of closed network connection Dec 28 15:17:46.791665625 (551h48m46.66274866s) [noise=true , address=64.49.13.169:4716 , piece_id=V3PN2MUBRFNCLCYOVHYLXUORQO4S2TLRL6HMGJQCMJ6PCBZISPKQ, seg_pos={0 0}, upload=483809]: Long tail cancellation due to optimal threshold Dec 28 15:17:46.791739836 (551h48m46.662822881s) [seg_pos={0 0}, upload=483809]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 15:17:46.791808337 (551h48m46.662891372s) [upload=483809]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 15:17:46.791812865 (551h48m46.6628959s) [upload=483809]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 15:17:47.097154206 (551h48m46.968237251s) [upload=483809]: Done uploading segments... Dec 28 15:17:47.097173253 (551h48m46.968256288s) [upload=483809]: Upload finished. err: <nil> Duration: 52.213224896s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 16.504822543s trace: https://jaeger.dev.storj.io/trace/4ce8174d0eaf2b7b FAIL (2025-12-28 15:22:46 UTC) orbiter warning: context deadline exceeded Duration: 16.504842622s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 42 out of 60 is failed (threshold=6) [XXXXXX.X..XXXXX.X.X.XXXXX...X.X.XX.XXX.X.XXXXX.XX.XX.X.XXXXX] duration: 5.001066157s trace: https://jaeger.dev.storj.io/trace/3df2a936138b1b9 FAIL (2025-12-28 15:22:37 UTC) orbiter warning: context deadline exceeded Duration: 5.001085653s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [...X...X.X.........X.....X......X.X.X.....X......XX.X..XXX.X] duration: 30.002458799s trace: https://jaeger.dev.storj.io/trace/210080892bd48307 FAIL (2025-12-28 15:22:24 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 15:21:55.690573314 (551h52m55.56165636s) [upload=483869]: Starting upload Dec 28 15:21:55.690634019 (551h52m55.561717064s) [upload=483869]: Done starting upload Dec 28 15:21:55.690705666 (551h52m55.561788711s) [upload=483869]: Uploading segments... Dec 28 15:21:55.690914573 (551h52m55.561997609s) [upload=483869]: Got next segment. Inline: false Dec 28 15:21:55.690946444 (551h52m55.562029489s) [upload=483869]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 15:21:55.690953247 (551h52m55.562036292s) [upload=483869]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 15:21:55.957013671 (551h52m55.828096706s) [seg_pos={0 0}, upload=483869]: Begin upload segment... Dec 28 15:21:55.957095196 (551h52m55.828178231s) [seg_pos={0 0}, upload=483869]: Done begin upload segment. Dec 28 15:21:55.958131419 (551h52m55.829214474s) [noise=true , address=145.79.227.4:4186 , piece_id=DUZI4UWTRCCHICNHAHN3BWJ3OOOM65UIYSLEYMV2J3SYD55S4PJQ, seg_pos={0 0}, upload=483869]: Uploading piece... Dec 28 15:21:55.959605496 (551h52m55.830688531s) [noise=true , address=145.79.227.11:4731 , piece_id=Q7GOF3OCVY7BC7MOJNKE7UJHJJVU57DTNUFJICMTTV7NDWIJG4XQ, seg_pos={0 0}, upload=483869]: Uploading piece... Dec 28 15:21:55.960168058 (551h52m55.831251093s) [noise=true , address=145.79.227.39:4015 , piece_id=VGXABVT3FVXSQQ7DHQWYSUE7S7UCAAP3TWERSTCZYGKNX5A64CWA, seg_pos={0 0}, upload=483869]: Uploading piece... Dec 28 15:21:55.962088605 (551h52m55.83317166s) [noise=true , address=45.91.116.6:4974 , piece_id=Q27YPVZKNCIYZRZKABJL7PM5JUYVTZPMADCN5N5ZJPR6AVJBLKFQ, seg_pos={0 0}, upload=483869]: Uploading piece... Dec 28 15:21:55.962692725 (551h52m55.83377576s) [noise=true , address=145.79.227.48:4302 , piece_id=CFN6LD43EF6IGY65CZU3HBYRTGNX46OKEU2XHG5AUUABMYYLZNZQ, seg_pos={0 0}, upload=483869]: Uploading piece... Dec 28 15:21:55.963247651 (551h52m55.834330686s) [noise=true , address=145.79.227.45:4231 , piece_id=V5Q76OWTPEPBSN6G25EDN6U7JK2EZPIFYU5GC5476AAEKADWO3OA, seg_pos={0 0}, upload=483869]: Uploading piece... Dec 28 15:21:55.963766640 (551h52m55.834849685s) [noise=true , address=45.91.116.5:4298 , piece_id=U3PHTR3AA2ZE2YJYFGZYKHQQMHEOWZFCGQO2PYBBK4IPX66SIA4Q, seg_pos={0 0}, upload=483869]: Uploading piece... Dec 28 15:21:55.964253447 (551h52m55.835336492s) [noise=true , address=145.79.227.36:4518 , piece_id=VQE5JSTUS5A4HWPSCS3DU2GXCZN5GZIGXGH2SJP7GEXNNIFSXKIQ, seg_pos={0 0}, upload=483869]: Uploading piece... Dec 28 15:21:55.964721898 (551h52m55.835804933s) [noise=true , address=145.79.227.46:4602 , piece_id=ZFXQTJOV5THQBEWRUVFNRF4ASX5V4LAZECS3LG2MSSHGIHR77STA, seg_pos={0 0}, upload=483869]: Uploading piece... Dec 28 15:21:55.965446218 (551h52m55.836529253s) [noise=true , address=77.90.31.13:4629 , piece_id=X3WSEQJXSZLPEN7LR4UCHDIBDWKOD6USQ7TNRYDI2YO4SC7PWCKQ, seg_pos={0 0}, upload=483869]: Uploading piece... Dec 28 15:21:55.968439207 (551h52m55.839522252s) [noise=true , address=45.91.116.7:4011 , piece_id=KZGXWS3H7RWVEIGZVNMBF3GWX2R7XA5HCZ6ZCHPHZRBQ4YWBQP7Q, seg_pos={0 0}, upload=483869]: Uploading piece... Dec 28 15:21:55.968848957 (551h52m55.839932002s) [noise=true , address=77.90.31.21:4878 , piece_id=S6EUHFVVYX6FXNSM24MYMTGVO4D5X3QVZZN2TCUYODMFTB4AWJAQ, seg_pos={0 0}, upload=483869]: Uploading piece... Dec 28 15:21:55.969454680 (551h52m55.840537725s) [noise=true , address=45.91.116.13:4153 , piece_id=IUUC2ALVV2APBGNPNFIUVWNFC6PAPGHOSHENGZX4VRUTCPEF6T3Q, seg_pos={0 0}, upload=483869]: Uploading piece... Dec 28 15:21:55.969854442 (551h52m55.840937477s) [noise=true , address=145.79.227.50:4373 , piece_id=ZHFPYVPMSFHWE56EZ6HYH35QMBU6FRCOH5PNPMRM7NYCMZIH6CHA, seg_pos={0 0}, upload=483869]: Uploading piece... Dec 28 15:21:55.978253365 (551h52m55.84933641s) [noise=true , address=45.91.116.11:4119 , piece_id=AOEN4PLL3Y6OOC6B4UDHSXSKE4AAZFENDW4MWJ4N77BCMAMBQTHQ, seg_pos={0 0}, upload=483869]: Uploading piece... Dec 28 15:21:55.985268051 (551h52m55.856351107s) [noise=true , address=145.79.227.41:4087 , piece_id=YN2MX7LEVJVR4JMPXFGMJGOL2DMJUEKI33LFNNBT2A7BN7LTLILA, seg_pos={0 0}, upload=483869]: Uploading piece... Dec 28 15:21:57.695500922 (551h52m57.566583967s) [upload=483869]: Next returned nil segment Dec 28 15:21:57.695519928 (551h52m57.566602993s) [upload=483869]: Waiting for error group managing segments... Dec 28 15:21:57.976465435 (551h52m57.84754848s) [noise=true , address=145.79.227.36:4518 , piece_id=VQE5JSTUS5A4HWPSCS3DU2GXCZN5GZIGXGH2SJP7GEXNNIFSXKIQ, seg_pos={0 0}, upload=483869]: Done uploading piece. err: <nil> Dec 28 15:21:57.980147786 (551h52m57.851230832s) [noise=true , address=145.79.227.50:4373 , piece_id=ZHFPYVPMSFHWE56EZ6HYH35QMBU6FRCOH5PNPMRM7NYCMZIH6CHA, seg_pos={0 0}, upload=483869]: Done uploading piece. err: <nil> Dec 28 15:21:57.980606651 (551h52m57.851689686s) [noise=true , address=145.79.227.4:4186 , piece_id=DUZI4UWTRCCHICNHAHN3BWJ3OOOM65UIYSLEYMV2J3SYD55S4PJQ, seg_pos={0 0}, upload=483869]: Done uploading piece. err: <nil> Dec 28 15:21:57.980669520 (551h52m57.851752555s) [noise=true , address=145.79.227.46:4602 , piece_id=ZFXQTJOV5THQBEWRUVFNRF4ASX5V4LAZECS3LG2MSSHGIHR77STA, seg_pos={0 0}, upload=483869]: Done uploading piece. err: <nil> Dec 28 15:21:57.981590334 (551h52m57.852673379s) [noise=true , address=145.79.227.48:4302 , piece_id=CFN6LD43EF6IGY65CZU3HBYRTGNX46OKEU2XHG5AUUABMYYLZNZQ, seg_pos={0 0}, upload=483869]: Done uploading piece. err: <nil> Dec 28 15:21:57.985324264 (551h52m57.856407309s) [noise=true , address=145.79.227.11:4731 , piece_id=Q7GOF3OCVY7BC7MOJNKE7UJHJJVU57DTNUFJICMTTV7NDWIJG4XQ, seg_pos={0 0}, upload=483869]: Done uploading piece. err: <nil> Dec 28 15:21:57.999049198 (551h52m57.870132233s) [noise=true , address=145.79.227.41:4087 , piece_id=YN2MX7LEVJVR4JMPXFGMJGOL2DMJUEKI33LFNNBT2A7BN7LTLILA, seg_pos={0 0}, upload=483869]: Done uploading piece. err: <nil> Dec 28 15:21:58.000320720 (551h52m57.871403775s) [noise=true , address=145.79.227.45:4231 , piece_id=V5Q76OWTPEPBSN6G25EDN6U7JK2EZPIFYU5GC5476AAEKADWO3OA, seg_pos={0 0}, upload=483869]: Done uploading piece. err: <nil> Dec 28 15:21:58.001534331 (551h52m57.872617386s) [noise=true , address=145.79.227.39:4015 , piece_id=VGXABVT3FVXSQQ7DHQWYSUE7S7UCAAP3TWERSTCZYGKNX5A64CWA, seg_pos={0 0}, upload=483869]: Done uploading piece. err: <nil> Dec 28 15:21:58.260468705 (551h52m58.13155175s) [noise=true , address=77.90.31.13:4629 , piece_id=X3WSEQJXSZLPEN7LR4UCHDIBDWKOD6USQ7TNRYDI2YO4SC7PWCKQ, seg_pos={0 0}, upload=483869]: Done uploading piece. err: <nil> Dec 28 15:21:58.348164904 (551h52m58.219247949s) [noise=true , address=45.91.116.6:4974 , piece_id=Q27YPVZKNCIYZRZKABJL7PM5JUYVTZPMADCN5N5ZJPR6AVJBLKFQ, seg_pos={0 0}, upload=483869]: Done uploading piece. err: <nil> Dec 28 15:21:58.356225171 (551h52m58.227308206s) [noise=true , address=45.91.116.11:4119 , piece_id=AOEN4PLL3Y6OOC6B4UDHSXSKE4AAZFENDW4MWJ4N77BCMAMBQTHQ, seg_pos={0 0}, upload=483869]: Done uploading piece. err: <nil> Dec 28 15:21:58.381959954 (551h52m58.253042989s) [noise=true , address=77.90.31.21:4878 , piece_id=S6EUHFVVYX6FXNSM24MYMTGVO4D5X3QVZZN2TCUYODMFTB4AWJAQ, seg_pos={0 0}, upload=483869]: Done uploading piece. err: <nil> Dec 28 15:21:58.381977728 (551h52m58.253060763s) [seg_pos={0 0}, upload=483869]: Segment reached optimal threshold of 13 pieces. Dec 28 15:21:58.382133495 (551h52m58.25321654s) [noise=true , address=45.91.116.7:4011 , piece_id=KZGXWS3H7RWVEIGZVNMBF3GWX2R7XA5HCZ6ZCHPHZRBQ4YWBQP7Q, seg_pos={0 0}, upload=483869]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12gZ5sVLg58D2gh3MVkHJ42qVRWrRU5EbuDWcG6hYU9MPgcSNQf): protocol: expected piece hash; context canceled; EOF Dec 28 15:21:58.382139386 (551h52m58.253222421s) [noise=true , address=45.91.116.7:4011 , piece_id=KZGXWS3H7RWVEIGZVNMBF3GWX2R7XA5HCZ6ZCHPHZRBQ4YWBQP7Q, seg_pos={0 0}, upload=483869]: Long tail cancellation due to optimal threshold Dec 28 15:21:58.382172429 (551h52m58.253255474s) [noise=true , address=45.91.116.5:4298 , piece_id=U3PHTR3AA2ZE2YJYFGZYKHQQMHEOWZFCGQO2PYBBK4IPX66SIA4Q, seg_pos={0 0}, upload=483869]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:39010->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:39010->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39010->45.91.116.5:4298: use of closed network connection Dec 28 15:21:58.382180163 (551h52m58.253263198s) [noise=true , address=45.91.116.5:4298 , piece_id=U3PHTR3AA2ZE2YJYFGZYKHQQMHEOWZFCGQO2PYBBK4IPX66SIA4Q, seg_pos={0 0}, upload=483869]: Long tail cancellation due to optimal threshold Dec 28 15:21:58.382266228 (551h52m58.253349263s) [noise=true , address=45.91.116.13:4153 , piece_id=IUUC2ALVV2APBGNPNFIUVWNFC6PAPGHOSHENGZX4VRUTCPEF6T3Q, seg_pos={0 0}, upload=483869]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b): protocol: expected piece hash; context canceled; EOF Dec 28 15:21:58.382273782 (551h52m58.253356837s) [noise=true , address=45.91.116.13:4153 , piece_id=IUUC2ALVV2APBGNPNFIUVWNFC6PAPGHOSHENGZX4VRUTCPEF6T3Q, seg_pos={0 0}, upload=483869]: Long tail cancellation due to optimal threshold Dec 28 15:21:58.382290523 (551h52m58.253373558s) [seg_pos={0 0}, upload=483869]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 15:21:58.382350488 (551h52m58.253433523s) [upload=483869]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 15:21:58.382356359 (551h52m58.253439404s) [upload=483869]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 15:21:58.726319273 (551h52m58.597402318s) [upload=483869]: Done uploading segments... Dec 28 15:21:58.726342005 (551h52m58.59742504s) [upload=483869]: Upload finished. err: <nil> Duration: 30.002479599s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 16.163559852s trace: https://jaeger.dev.storj.io/trace/4ce8174d0eaf2b7b FAIL (2025-12-28 15:22:10 UTC) orbiter warning: context deadline exceeded Duration: 16.163589187s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.......X......X....X..........................X....X...X....] duration: 42.350441106s trace: https://jaeger.dev.storj.io/trace/51e2216a4efa8b95 FAIL (2025-12-28 15:21:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 15s) [noise=true , address=145.79.227.40:4083 , piece_id=RSMZDEGC4MX5TFRFKI33WNDK42NZDQ4M6UPRQWO4GE6VWGCJ6AFA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.854006008 (551h48m43.725089043s) [noise=true , address=77.90.31.12:4610 , piece_id=RSI6EFWFTX4JZUBEFTV3VPMJWMWI7VDMUHG6DMK7CSKTH7MDFGXA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.854333892 (551h48m43.725416937s) [noise=true , address=64.49.13.139:4476 , piece_id=FUSCXA3KPS24G3R5ORJTDOUOU2KMIK5SNBPROHL2CRKMKBRWFN6A, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.854753131 (551h48m43.725836176s) [noise=true , address=145.79.227.34:4932 , piece_id=5NGLBPM6NRZIR4NJTA43KIFQQTD6DS6B37DIFCHXGT5YMDGNMXKA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.855102245 (551h48m43.726185381s) [noise=true , address=77.90.31.15:4727 , piece_id=YJDXXQTMZQRYCEAKLZUFPW6JO7RWHJMJAOFMURYFNLQSJ4LJG2CQ, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.855467040 (551h48m43.726550076s) [noise=true , address=145.79.227.6:4280 , piece_id=ARMYEAFMP6DPW2JSHWHOYPJ6OUUSUQETJCSSQQXFQX4DDCVZYLYQ, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.855834780 (551h48m43.726917815s) [noise=true , address=193.149.164.143:4694 , piece_id=SDK5DIHWVFOZQIHQWJMZLOTQJEKRDBUTQ7R7ZTKOKF5LGECJRMOA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.856263167 (551h48m43.727346212s) [noise=true , address=145.79.227.43:4167 , piece_id=WEXP7LDX6O7KPXA6XIMRWNTMGNGSTE3XRFODPNMWFKR3D5WXX3LQ, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.858002919 (551h48m43.729085964s) [noise=true , address=145.79.227.30:4766 , piece_id=YA4HOOMXQRQCPE3VMQ6WEXBM3ZIED4RZLVFHZHRLBQJFQBFP2F7A, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.858063876 (551h48m43.729146911s) [noise=true , address=64.49.13.169:4716 , piece_id=V3PN2MUBRFNCLCYOVHYLXUORQO4S2TLRL6HMGJQCMJ6PCBZISPKQ, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.859819067 (551h48m43.730902113s) [noise=true , address=145.79.227.49:4367 , piece_id=CITJCIKXK4LYSUV6QRTLIHB7SEH5QOPUM7XR4JDHV54EJ5FLLMDQ, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.861538431 (551h48m43.732621466s) [noise=true , address=145.79.227.33:4904 , piece_id=4B64RHS2HQ6WZYND6MZGU4AZC4LQBXPNKOE2ABNVZQTEBXALQCDA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.863179095 (551h48m43.73426214s) [noise=true , address=77.90.31.19:4874 , piece_id=IXOOASSSDYJS42HWYZKK4PKE4LW47BT4WF5WRRXY7ANPLY632LRA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.864788100 (551h48m43.735871145s) [noise=true , address=77.90.31.16:4749 , piece_id=IVHKFK76TGLMJ7IBVS5ZCCMB35LKHF4FE5HLE7P5QAVGAWGSRHNA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.866240675 (551h48m43.73732371s) [noise=true , address=45.91.116.7:4029 , piece_id=VTKGXA2SWJZHOLLWREHQO66OHQZWHVRK25STFR62PEIJFGD4SG4A, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.867523768 (551h48m43.738606803s) [noise=true , address=45.91.116.6:4998 , piece_id=KCEYZDX2WGV7UCYX4H4BNSR2CWOHGPSEW76LKU2BGC3UDHA5M5SA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:45.143085883 (551h48m45.014168928s) [upload=483809]: Next returned nil segment Dec 28 15:17:45.143099779 (551h48m45.014182824s) [upload=483809]: Waiting for error group managing segments... Dec 28 15:17:45.472957433 (551h48m45.344040468s) [noise=true , address=64.49.13.139:4476 , piece_id=FUSCXA3KPS24G3R5ORJTDOUOU2KMIK5SNBPROHL2CRKMKBRWFN6A, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:45.739300305 (551h48m45.61038334s) [noise=true , address=145.79.227.6:4280 , piece_id=ARMYEAFMP6DPW2JSHWHOYPJ6OUUSUQETJCSSQQXFQX4DDCVZYLYQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:45.743884264 (551h48m45.614967309s) [noise=true , address=145.79.227.21:4508 , piece_id=PGAOV6UBGJPU7M5L3IR2LQU5LSISUJVKKBB7ZRV233F5MBEAWK6A, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:45.936004292 (551h48m45.807087328s) [noise=true , address=77.90.31.15:4727 , piece_id=YJDXXQTMZQRYCEAKLZUFPW6JO7RWHJMJAOFMURYFNLQSJ4LJG2CQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:45.968797306 (551h48m45.839880351s) [noise=true , address=77.90.31.16:4749 , piece_id=IVHKFK76TGLMJ7IBVS5ZCCMB35LKHF4FE5HLE7P5QAVGAWGSRHNA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.007515762 (551h48m45.878598797s) [noise=true , address=77.90.31.11:4580 , piece_id=UUPDJ75SL3HOJREC2SXJGQMUVLLP5WLN2L3LEON6KIKPB24DEKPQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.147397009 (551h48m46.018480054s) [noise=true , address=45.91.116.7:4029 , piece_id=VTKGXA2SWJZHOLLWREHQO66OHQZWHVRK25STFR62PEIJFGD4SG4A, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.460756007 (551h48m46.331839042s) [noise=true , address=193.149.164.143:4694 , piece_id=SDK5DIHWVFOZQIHQWJMZLOTQJEKRDBUTQ7R7ZTKOKF5LGECJRMOA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.662946081 (551h48m46.534029127s) [noise=true , address=145.79.227.43:4167 , piece_id=WEXP7LDX6O7KPXA6XIMRWNTMGNGSTE3XRFODPNMWFKR3D5WXX3LQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.695276604 (551h48m46.566359639s) [noise=true , address=45.91.116.6:4998 , piece_id=KCEYZDX2WGV7UCYX4H4BNSR2CWOHGPSEW76LKU2BGC3UDHA5M5SA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.739405984 (551h48m46.610489029s) [noise=true , address=145.79.227.33:4904 , piece_id=4B64RHS2HQ6WZYND6MZGU4AZC4LQBXPNKOE2ABNVZQTEBXALQCDA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.764293434 (551h48m46.635376479s) [noise=true , address=145.79.227.40:4083 , piece_id=RSMZDEGC4MX5TFRFKI33WNDK42NZDQ4M6UPRQWO4GE6VWGCJ6AFA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.771299524 (551h48m46.64238256s) [noise=true , address=145.79.227.30:4766 , piece_id=YA4HOOMXQRQCPE3VMQ6WEXBM3ZIED4RZLVFHZHRLBQJFQBFP2F7A, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.791263379 (551h48m46.662346414s) [noise=true , address=77.90.31.19:4874 , piece_id=IXOOASSSDYJS42HWYZKK4PKE4LW47BT4WF5WRRXY7ANPLY632LRA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.791282065 (551h48m46.6623651s) [seg_pos={0 0}, upload=483809]: Segment reached optimal threshold of 14 pieces. Dec 28 15:17:46.791451286 (551h48m46.662534321s) [noise=true , address=145.79.227.49:4367 , piece_id=CITJCIKXK4LYSUV6QRTLIHB7SEH5QOPUM7XR4JDHV54EJ5FLLMDQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF Dec 28 15:17:46.791458450 (551h48m46.662541496s) [noise=true , address=145.79.227.49:4367 , piece_id=CITJCIKXK4LYSUV6QRTLIHB7SEH5QOPUM7XR4JDHV54EJ5FLLMDQ, seg_pos={0 0}, upload=483809]: Long tail cancellation due to optimal threshold Dec 28 15:17:46.791491023 (551h48m46.662574058s) [noise=true , address=77.90.31.12:4610 , piece_id=RSI6EFWFTX4JZUBEFTV3VPMJWMWI7VDMUHG6DMK7CSKTH7MDFGXA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi): protocol: expected piece hash; context canceled; EOF Dec 28 15:17:46.791502935 (551h48m46.66258598s) [noise=true , address=77.90.31.12:4610 , piece_id=RSI6EFWFTX4JZUBEFTV3VPMJWMWI7VDMUHG6DMK7CSKTH7MDFGXA, seg_pos={0 0}, upload=483809]: Long tail cancellation due to optimal threshold Dec 28 15:17:46.791556076 (551h48m46.662639111s) [noise=true , address=145.79.227.34:4932 , piece_id=5NGLBPM6NRZIR4NJTA43KIFQQTD6DS6B37DIFCHXGT5YMDGNMXKA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qJfPZErJx4XGBw7NmfCLS7BogRnN29rUibuWkzQDFUNvPBpmm): protocol: expected piece hash; context canceled; EOF Dec 28 15:17:46.791561086 (551h48m46.662644131s) [noise=true , address=145.79.227.34:4932 , piece_id=5NGLBPM6NRZIR4NJTA43KIFQQTD6DS6B37DIFCHXGT5YMDGNMXKA, seg_pos={0 0}, upload=483809]: Long tail cancellation due to optimal threshold Dec 28 15:17:46.791659873 (551h48m46.662742908s) [noise=true , address=64.49.13.169:4716 , piece_id=V3PN2MUBRFNCLCYOVHYLXUORQO4S2TLRL6HMGJQCMJ6PCBZISPKQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:48986->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:48986->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48986->64.49.13.169:4716: use of closed network connection Dec 28 15:17:46.791665625 (551h48m46.66274866s) [noise=true , address=64.49.13.169:4716 , piece_id=V3PN2MUBRFNCLCYOVHYLXUORQO4S2TLRL6HMGJQCMJ6PCBZISPKQ, seg_pos={0 0}, upload=483809]: Long tail cancellation due to optimal threshold Dec 28 15:17:46.791739836 (551h48m46.662822881s) [seg_pos={0 0}, upload=483809]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 15:17:46.791808337 (551h48m46.662891372s) [upload=483809]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 15:17:46.791812865 (551h48m46.6628959s) [upload=483809]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 15:17:47.097154206 (551h48m46.968237251s) [upload=483809]: Done uploading segments... Dec 28 15:17:47.097173253 (551h48m46.968256288s) [upload=483809]: Upload finished. err: <nil> Duration: 42.350465483s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 42 out of 60 is failed (threshold=6) [XXXXXXX.X..XXXXX.X.X.XXXXX...X.X.XX.XXX.X.XXXXX.XX.XX.X.XXXX] duration: 5.001289453s trace: https://jaeger.dev.storj.io/trace/1a73f2f96c4b2ff5 FAIL (2025-12-28 15:21:38 UTC) orbiter warning: context deadline exceeded Duration: 5.001317566s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [X...X...X.X.........X.....X......X.X.X.....X......XX.X..XXX.] duration: 6.194159785s trace: https://jaeger.dev.storj.io/trace/30f47d2876676052 FAIL (2025-12-28 15:21:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 15:20:40.698249766 (551h51m40.569332801s) [upload=483852]: Starting upload Dec 28 15:20:40.698332093 (551h51m40.569415128s) [upload=483852]: Done starting upload Dec 28 15:20:40.698376297 (551h51m40.569459332s) [upload=483852]: Uploading segments... Dec 28 15:20:40.698641382 (551h51m40.569724417s) [upload=483852]: Got next segment. Inline: false Dec 28 15:20:40.698688581 (551h51m40.569771636s) [upload=483852]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 15:20:40.698695695 (551h51m40.56977874s) [upload=483852]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 15:20:40.949316900 (551h51m40.820399935s) [seg_pos={0 0}, upload=483852]: Begin upload segment... Dec 28 15:20:40.949373357 (551h51m40.820456392s) [seg_pos={0 0}, upload=483852]: Done begin upload segment. Dec 28 15:20:40.949412391 (551h51m40.820495426s) [noise=true , address=145.79.227.44:4193 , piece_id=M2XNJ2OSDTN23RNRZ4X5QI7VTFKEL7ZKSY4SWKTLK7ZU3BW2FE3Q, seg_pos={0 0}, upload=483852]: Uploading piece... Dec 28 15:20:40.949435987 (551h51m40.820519042s) [noise=true , address=145.79.227.2:4115 , piece_id=LIRQF4HCLA265YZ4HMMFNKDWAJG5ALYSMMUTHZ7FVU4K7LS4IMCQ, seg_pos={0 0}, upload=483852]: Uploading piece... Dec 28 15:20:40.950151209 (551h51m40.821234254s) [noise=true , address=145.79.227.21:4473 , piece_id=6WSGGPWMXQF6PUQMHHW7EFYFQCRYLVLM7YGIQ27RSFJTM2HQNIDQ, seg_pos={0 0}, upload=483852]: Uploading piece... Dec 28 15:20:40.950684574 (551h51m40.821767629s) [noise=true , address=145.79.227.39:4015 , piece_id=OPYSB2OFSGH6Y4VKAVICQ6FLNUA5SKV3MH36RDXFBAAUPW3ET47Q, seg_pos={0 0}, upload=483852]: Uploading piece... Dec 28 15:20:40.951091018 (551h51m40.822174053s) [noise=true , address=45.91.116.13:4155 , piece_id=IR7SXMMFR53EUIAN2YZAQH5XGJ3AENNRSNWP7HFF6UANZ67ZPR6A, seg_pos={0 0}, upload=483852]: Uploading piece... Dec 28 15:20:40.951487393 (551h51m40.822570438s) [noise=true , address=145.79.227.11:4730 , piece_id=FDXILLKXJHJJIAKWVD7D5ZE5ES2A7DDMZWSZYT4PML3CRY2Z26BA, seg_pos={0 0}, upload=483852]: Uploading piece... Dec 28 15:20:40.951908475 (551h51m40.82299152s) [noise=true , address=145.79.227.50:4374 , piece_id=EI2YJUVWQC4FCJPZ6GSQARWZTEJQVT6J62APQGZE3B4REPWPQ5XA, seg_pos={0 0}, upload=483852]: Uploading piece... Dec 28 15:20:40.952332622 (551h51m40.823415667s) [noise=true , address=45.91.116.10:4084 , piece_id=4OGYZJYB5G3ZZYWDRCNFRPEREMC2BMTZVIYQFCVNDC3DTJU4Q55A, seg_pos={0 0}, upload=483852]: Uploading piece... Dec 28 15:20:40.953595588 (551h51m40.824678623s) [noise=true , address=145.79.227.35:4483 , piece_id=H5JUJ4B54KE57CX3AIQENSZ5TNZGEL4KWDIQD5X4AEZLMXG3ORGQ, seg_pos={0 0}, upload=483852]: Uploading piece... Dec 28 15:20:40.955013637 (551h51m40.826096672s) [noise=true , address=77.90.31.16:4736 , piece_id=UR34NNCT6QSWZFG7VIMQREBEJM4FAO7U5W2C354XGBDLZJHNZ7YA, seg_pos={0 0}, upload=483852]: Uploading piece... Dec 28 15:20:40.956888978 (551h51m40.827972023s) [noise=true , address=145.79.227.4:4186 , piece_id=CHZQC54AMZKCRMBY3DXHOPDBTHUNU33ANHPCXOAJSQDXLFGULEGA, seg_pos={0 0}, upload=483852]: Uploading piece... Dec 28 15:20:40.958971895 (551h51m40.83005494s) [noise=true , address=145.79.227.13:4804 , piece_id=ZEXXTYW45SAWL6BFAQKHM3W66MLLM4SEKQGOD65NPXTGL6XJPADQ, seg_pos={0 0}, upload=483852]: Uploading piece... Dec 28 15:20:40.960429390 (551h51m40.831512425s) [noise=true , address=145.79.227.17:4972 , piece_id=4MVJDEAEB5FGHIF6QII7CBHCKL3CDVVSBHKZJXLEJ3LVO3JISU6A, seg_pos={0 0}, upload=483852]: Uploading piece... Dec 28 15:20:40.961475482 (551h51m40.832558527s) [noise=true , address=45.91.116.7:4009 , piece_id=W3A24DGDIV6HTEANA6OBOVVLSHLLUVRIXF2MLBO2HBW52JWEBFDA, seg_pos={0 0}, upload=483852]: Uploading piece... Dec 28 15:20:40.962576799 (551h51m40.833659834s) [noise=true , address=145.79.227.16:4911 , piece_id=6WZ3CHAZQAJA2ULJ43FNBTOPTGATKHW552VWF4UN5A7LHIMG7Q2Q, seg_pos={0 0}, upload=483852]: Uploading piece... Dec 28 15:20:40.963561214 (551h51m40.834644259s) [noise=true , address=145.79.227.6:4259 , piece_id=5RVNG2ALZSNRA4CPC6PW6A75LKEOEXJUVUS3A4RK5N4XSJU75XEA, seg_pos={0 0}, upload=483852]: Uploading piece... Dec 28 15:20:42.667422049 (551h51m42.538505114s) [upload=483852]: Next returned nil segment Dec 28 15:20:42.667452538 (551h51m42.538535613s) [upload=483852]: Waiting for error group managing segments... Dec 28 15:20:42.953594740 (551h51m42.824677785s) [noise=true , address=145.79.227.2:4115 , piece_id=LIRQF4HCLA265YZ4HMMFNKDWAJG5ALYSMMUTHZ7FVU4K7LS4IMCQ, seg_pos={0 0}, upload=483852]: Done uploading piece. err: <nil> Dec 28 15:20:42.953651709 (551h51m42.824734764s) [noise=true , address=145.79.227.11:4730 , piece_id=FDXILLKXJHJJIAKWVD7D5ZE5ES2A7DDMZWSZYT4PML3CRY2Z26BA, seg_pos={0 0}, upload=483852]: Done uploading piece. err: <nil> Dec 28 15:20:42.954391397 (551h51m42.825474442s) [noise=true , address=145.79.227.39:4015 , piece_id=OPYSB2OFSGH6Y4VKAVICQ6FLNUA5SKV3MH36RDXFBAAUPW3ET47Q, seg_pos={0 0}, upload=483852]: Done uploading piece. err: <nil> Dec 28 15:20:42.962501069 (551h51m42.833584104s) [noise=true , address=145.79.227.17:4972 , piece_id=4MVJDEAEB5FGHIF6QII7CBHCKL3CDVVSBHKZJXLEJ3LVO3JISU6A, seg_pos={0 0}, upload=483852]: Done uploading piece. err: <nil> Dec 28 15:20:42.965706503 (551h51m42.836789538s) [noise=true , address=145.79.227.13:4804 , piece_id=ZEXXTYW45SAWL6BFAQKHM3W66MLLM4SEKQGOD65NPXTGL6XJPADQ, seg_pos={0 0}, upload=483852]: Done uploading piece. err: <nil> Dec 28 15:20:42.969637208 (551h51m42.840720243s) [noise=true , address=145.79.227.16:4911 , piece_id=6WZ3CHAZQAJA2ULJ43FNBTOPTGATKHW552VWF4UN5A7LHIMG7Q2Q, seg_pos={0 0}, upload=483852]: Done uploading piece. err: <nil> Dec 28 15:20:42.975387547 (551h51m42.846470582s) [noise=true , address=145.79.227.50:4374 , piece_id=EI2YJUVWQC4FCJPZ6GSQARWZTEJQVT6J62APQGZE3B4REPWPQ5XA, seg_pos={0 0}, upload=483852]: Done uploading piece. err: <nil> Dec 28 15:20:42.977822744 (551h51m42.848905789s) [noise=true , address=145.79.227.21:4473 , piece_id=6WSGGPWMXQF6PUQMHHW7EFYFQCRYLVLM7YGIQ27RSFJTM2HQNIDQ, seg_pos={0 0}, upload=483852]: Done uploading piece. err: <nil> Dec 28 15:20:42.987920853 (551h51m42.859003888s) [noise=true , address=145.79.227.6:4259 , piece_id=5RVNG2ALZSNRA4CPC6PW6A75LKEOEXJUVUS3A4RK5N4XSJU75XEA, seg_pos={0 0}, upload=483852]: Done uploading piece. err: <nil> Dec 28 15:20:42.988629783 (551h51m42.859712818s) [noise=true , address=145.79.227.4:4186 , piece_id=CHZQC54AMZKCRMBY3DXHOPDBTHUNU33ANHPCXOAJSQDXLFGULEGA, seg_pos={0 0}, upload=483852]: Done uploading piece. err: <nil> Dec 28 15:20:43.175271810 (551h51m43.046354845s) [noise=true , address=45.91.116.7:4009 , piece_id=W3A24DGDIV6HTEANA6OBOVVLSHLLUVRIXF2MLBO2HBW52JWEBFDA, seg_pos={0 0}, upload=483852]: Done uploading piece. err: <nil> Dec 28 15:20:43.215866780 (551h51m43.086949815s) [noise=true , address=45.91.116.13:4155 , piece_id=IR7SXMMFR53EUIAN2YZAQH5XGJ3AENNRSNWP7HFF6UANZ67ZPR6A, seg_pos={0 0}, upload=483852]: Done uploading piece. err: <nil> Dec 28 15:20:43.281784138 (551h51m43.152867173s) [noise=true , address=45.91.116.10:4084 , piece_id=4OGYZJYB5G3ZZYWDRCNFRPEREMC2BMTZVIYQFCVNDC3DTJU4Q55A, seg_pos={0 0}, upload=483852]: Done uploading piece. err: <nil> Dec 28 15:20:43.281836027 (551h51m43.152919062s) [seg_pos={0 0}, upload=483852]: Segment reached optimal threshold of 13 pieces. Dec 28 15:20:43.282052929 (551h51m43.153135964s) [noise=true , address=145.79.227.35:4483 , piece_id=H5JUJ4B54KE57CX3AIQENSZ5TNZGEL4KWDIQD5X4AEZLMXG3ORGQ, seg_pos={0 0}, upload=483852]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 28 15:20:43.282060634 (551h51m43.153143669s) [noise=true , address=145.79.227.35:4483 , piece_id=H5JUJ4B54KE57CX3AIQENSZ5TNZGEL4KWDIQD5X4AEZLMXG3ORGQ, seg_pos={0 0}, upload=483852]: Long tail cancellation due to optimal threshold Dec 28 15:20:43.282235757 (551h51m43.153318792s) [noise=true , address=77.90.31.16:4736 , piece_id=UR34NNCT6QSWZFG7VIMQREBEJM4FAO7U5W2C354XGBDLZJHNZ7YA, seg_pos={0 0}, upload=483852]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 28 15:20:43.282252710 (551h51m43.153335745s) [noise=true , address=77.90.31.16:4736 , piece_id=UR34NNCT6QSWZFG7VIMQREBEJM4FAO7U5W2C354XGBDLZJHNZ7YA, seg_pos={0 0}, upload=483852]: Long tail cancellation due to optimal threshold Dec 28 15:20:43.282317734 (551h51m43.153400769s) [noise=true , address=145.79.227.44:4193 , piece_id=M2XNJ2OSDTN23RNRZ4X5QI7VTFKEL7ZKSY4SWKTLK7ZU3BW2FE3Q, seg_pos={0 0}, upload=483852]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 28 15:20:43.282325318 (551h51m43.153408363s) [noise=true , address=145.79.227.44:4193 , piece_id=M2XNJ2OSDTN23RNRZ4X5QI7VTFKEL7ZKSY4SWKTLK7ZU3BW2FE3Q, seg_pos={0 0}, upload=483852]: Long tail cancellation due to optimal threshold Dec 28 15:20:43.282367418 (551h51m43.153450453s) [seg_pos={0 0}, upload=483852]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 15:20:43.282497877 (551h51m43.153580912s) [upload=483852]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 15:20:43.282503738 (551h51m43.153586773s) [upload=483852]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 15:20:43.579155893 (551h51m43.450238938s) [upload=483852]: Done uploading segments... Dec 28 15:20:43.579182033 (551h51m43.450265078s) [upload=483852]: Upload finished. err: <nil> Duration: 6.194258363s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 14.326585344s trace: https://jaeger.dev.storj.io/trace/70ba9ccd98ccb34e FAIL (2025-12-28 15:21:30 UTC) orbiter warning: context deadline exceeded Duration: 14.326609509s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.X...X...X.X.........X.....X......X.X.X.....X......XX.X..XXX] duration: 30.003962699s trace: https://jaeger.dev.storj.io/trace/30f47d2876676052 FAIL (2025-12-28 15:21:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 15:20:40.698249766 (551h51m40.569332801s) [upload=483852]: Starting upload Dec 28 15:20:40.698332093 (551h51m40.569415128s) [upload=483852]: Done starting upload Dec 28 15:20:40.698376297 (551h51m40.569459332s) [upload=483852]: Uploading segments... Dec 28 15:20:40.698641382 (551h51m40.569724417s) [upload=483852]: Got next segment. Inline: false Dec 28 15:20:40.698688581 (551h51m40.569771636s) [upload=483852]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 15:20:40.698695695 (551h51m40.56977874s) [upload=483852]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 15:20:40.949316900 (551h51m40.820399935s) [seg_pos={0 0}, upload=483852]: Begin upload segment... Dec 28 15:20:40.949373357 (551h51m40.820456392s) [seg_pos={0 0}, upload=483852]: Done begin upload segment. Dec 28 15:20:40.949412391 (551h51m40.820495426s) [noise=true , address=145.79.227.44:4193 , piece_id=M2XNJ2OSDTN23RNRZ4X5QI7VTFKEL7ZKSY4SWKTLK7ZU3BW2FE3Q, seg_pos={0 0}, upload=483852]: Uploading piece... Dec 28 15:20:40.949435987 (551h51m40.820519042s) [noise=true , address=145.79.227.2:4115 , piece_id=LIRQF4HCLA265YZ4HMMFNKDWAJG5ALYSMMUTHZ7FVU4K7LS4IMCQ, seg_pos={0 0}, upload=483852]: Uploading piece... Dec 28 15:20:40.950151209 (551h51m40.821234254s) [noise=true , address=145.79.227.21:4473 , piece_id=6WSGGPWMXQF6PUQMHHW7EFYFQCRYLVLM7YGIQ27RSFJTM2HQNIDQ, seg_pos={0 0}, upload=483852]: Uploading piece... Dec 28 15:20:40.950684574 (551h51m40.821767629s) [noise=true , address=145.79.227.39:4015 , piece_id=OPYSB2OFSGH6Y4VKAVICQ6FLNUA5SKV3MH36RDXFBAAUPW3ET47Q, seg_pos={0 0}, upload=483852]: Uploading piece... Dec 28 15:20:40.951091018 (551h51m40.822174053s) [noise=true , address=45.91.116.13:4155 , piece_id=IR7SXMMFR53EUIAN2YZAQH5XGJ3AENNRSNWP7HFF6UANZ67ZPR6A, seg_pos={0 0}, upload=483852]: Uploading piece... Dec 28 15:20:40.951487393 (551h51m40.822570438s) [noise=true , address=145.79.227.11:4730 , piece_id=FDXILLKXJHJJIAKWVD7D5ZE5ES2A7DDMZWSZYT4PML3CRY2Z26BA, seg_pos={0 0}, upload=483852]: Uploading piece... Dec 28 15:20:40.951908475 (551h51m40.82299152s) [noise=true , address=145.79.227.50:4374 , piece_id=EI2YJUVWQC4FCJPZ6GSQARWZTEJQVT6J62APQGZE3B4REPWPQ5XA, seg_pos={0 0}, upload=483852]: Uploading piece... Dec 28 15:20:40.952332622 (551h51m40.823415667s) [noise=true , address=45.91.116.10:4084 , piece_id=4OGYZJYB5G3ZZYWDRCNFRPEREMC2BMTZVIYQFCVNDC3DTJU4Q55A, seg_pos={0 0}, upload=483852]: Uploading piece... Dec 28 15:20:40.953595588 (551h51m40.824678623s) [noise=true , address=145.79.227.35:4483 , piece_id=H5JUJ4B54KE57CX3AIQENSZ5TNZGEL4KWDIQD5X4AEZLMXG3ORGQ, seg_pos={0 0}, upload=483852]: Uploading piece... Dec 28 15:20:40.955013637 (551h51m40.826096672s) [noise=true , address=77.90.31.16:4736 , piece_id=UR34NNCT6QSWZFG7VIMQREBEJM4FAO7U5W2C354XGBDLZJHNZ7YA, seg_pos={0 0}, upload=483852]: Uploading piece... Dec 28 15:20:40.956888978 (551h51m40.827972023s) [noise=true , address=145.79.227.4:4186 , piece_id=CHZQC54AMZKCRMBY3DXHOPDBTHUNU33ANHPCXOAJSQDXLFGULEGA, seg_pos={0 0}, upload=483852]: Uploading piece... Dec 28 15:20:40.958971895 (551h51m40.83005494s) [noise=true , address=145.79.227.13:4804 , piece_id=ZEXXTYW45SAWL6BFAQKHM3W66MLLM4SEKQGOD65NPXTGL6XJPADQ, seg_pos={0 0}, upload=483852]: Uploading piece... Dec 28 15:20:40.960429390 (551h51m40.831512425s) [noise=true , address=145.79.227.17:4972 , piece_id=4MVJDEAEB5FGHIF6QII7CBHCKL3CDVVSBHKZJXLEJ3LVO3JISU6A, seg_pos={0 0}, upload=483852]: Uploading piece... Dec 28 15:20:40.961475482 (551h51m40.832558527s) [noise=true , address=45.91.116.7:4009 , piece_id=W3A24DGDIV6HTEANA6OBOVVLSHLLUVRIXF2MLBO2HBW52JWEBFDA, seg_pos={0 0}, upload=483852]: Uploading piece... Dec 28 15:20:40.962576799 (551h51m40.833659834s) [noise=true , address=145.79.227.16:4911 , piece_id=6WZ3CHAZQAJA2ULJ43FNBTOPTGATKHW552VWF4UN5A7LHIMG7Q2Q, seg_pos={0 0}, upload=483852]: Uploading piece... Dec 28 15:20:40.963561214 (551h51m40.834644259s) [noise=true , address=145.79.227.6:4259 , piece_id=5RVNG2ALZSNRA4CPC6PW6A75LKEOEXJUVUS3A4RK5N4XSJU75XEA, seg_pos={0 0}, upload=483852]: Uploading piece... Dec 28 15:20:42.667422049 (551h51m42.538505114s) [upload=483852]: Next returned nil segment Dec 28 15:20:42.667452538 (551h51m42.538535613s) [upload=483852]: Waiting for error group managing segments... Dec 28 15:20:42.953594740 (551h51m42.824677785s) [noise=true , address=145.79.227.2:4115 , piece_id=LIRQF4HCLA265YZ4HMMFNKDWAJG5ALYSMMUTHZ7FVU4K7LS4IMCQ, seg_pos={0 0}, upload=483852]: Done uploading piece. err: <nil> Dec 28 15:20:42.953651709 (551h51m42.824734764s) [noise=true , address=145.79.227.11:4730 , piece_id=FDXILLKXJHJJIAKWVD7D5ZE5ES2A7DDMZWSZYT4PML3CRY2Z26BA, seg_pos={0 0}, upload=483852]: Done uploading piece. err: <nil> Dec 28 15:20:42.954391397 (551h51m42.825474442s) [noise=true , address=145.79.227.39:4015 , piece_id=OPYSB2OFSGH6Y4VKAVICQ6FLNUA5SKV3MH36RDXFBAAUPW3ET47Q, seg_pos={0 0}, upload=483852]: Done uploading piece. err: <nil> Dec 28 15:20:42.962501069 (551h51m42.833584104s) [noise=true , address=145.79.227.17:4972 , piece_id=4MVJDEAEB5FGHIF6QII7CBHCKL3CDVVSBHKZJXLEJ3LVO3JISU6A, seg_pos={0 0}, upload=483852]: Done uploading piece. err: <nil> Dec 28 15:20:42.965706503 (551h51m42.836789538s) [noise=true , address=145.79.227.13:4804 , piece_id=ZEXXTYW45SAWL6BFAQKHM3W66MLLM4SEKQGOD65NPXTGL6XJPADQ, seg_pos={0 0}, upload=483852]: Done uploading piece. err: <nil> Dec 28 15:20:42.969637208 (551h51m42.840720243s) [noise=true , address=145.79.227.16:4911 , piece_id=6WZ3CHAZQAJA2ULJ43FNBTOPTGATKHW552VWF4UN5A7LHIMG7Q2Q, seg_pos={0 0}, upload=483852]: Done uploading piece. err: <nil> Dec 28 15:20:42.975387547 (551h51m42.846470582s) [noise=true , address=145.79.227.50:4374 , piece_id=EI2YJUVWQC4FCJPZ6GSQARWZTEJQVT6J62APQGZE3B4REPWPQ5XA, seg_pos={0 0}, upload=483852]: Done uploading piece. err: <nil> Dec 28 15:20:42.977822744 (551h51m42.848905789s) [noise=true , address=145.79.227.21:4473 , piece_id=6WSGGPWMXQF6PUQMHHW7EFYFQCRYLVLM7YGIQ27RSFJTM2HQNIDQ, seg_pos={0 0}, upload=483852]: Done uploading piece. err: <nil> Dec 28 15:20:42.987920853 (551h51m42.859003888s) [noise=true , address=145.79.227.6:4259 , piece_id=5RVNG2ALZSNRA4CPC6PW6A75LKEOEXJUVUS3A4RK5N4XSJU75XEA, seg_pos={0 0}, upload=483852]: Done uploading piece. err: <nil> Dec 28 15:20:42.988629783 (551h51m42.859712818s) [noise=true , address=145.79.227.4:4186 , piece_id=CHZQC54AMZKCRMBY3DXHOPDBTHUNU33ANHPCXOAJSQDXLFGULEGA, seg_pos={0 0}, upload=483852]: Done uploading piece. err: <nil> Dec 28 15:20:43.175271810 (551h51m43.046354845s) [noise=true , address=45.91.116.7:4009 , piece_id=W3A24DGDIV6HTEANA6OBOVVLSHLLUVRIXF2MLBO2HBW52JWEBFDA, seg_pos={0 0}, upload=483852]: Done uploading piece. err: <nil> Dec 28 15:20:43.215866780 (551h51m43.086949815s) [noise=true , address=45.91.116.13:4155 , piece_id=IR7SXMMFR53EUIAN2YZAQH5XGJ3AENNRSNWP7HFF6UANZ67ZPR6A, seg_pos={0 0}, upload=483852]: Done uploading piece. err: <nil> Dec 28 15:20:43.281784138 (551h51m43.152867173s) [noise=true , address=45.91.116.10:4084 , piece_id=4OGYZJYB5G3ZZYWDRCNFRPEREMC2BMTZVIYQFCVNDC3DTJU4Q55A, seg_pos={0 0}, upload=483852]: Done uploading piece. err: <nil> Dec 28 15:20:43.281836027 (551h51m43.152919062s) [seg_pos={0 0}, upload=483852]: Segment reached optimal threshold of 13 pieces. Dec 28 15:20:43.282052929 (551h51m43.153135964s) [noise=true , address=145.79.227.35:4483 , piece_id=H5JUJ4B54KE57CX3AIQENSZ5TNZGEL4KWDIQD5X4AEZLMXG3ORGQ, seg_pos={0 0}, upload=483852]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1igSs2ZGX3zWVsSdmm1pnwYd7JckcyBubTuNU1gJDxQsZ9nKaY): protocol: expected piece hash; context canceled; EOF Dec 28 15:20:43.282060634 (551h51m43.153143669s) [noise=true , address=145.79.227.35:4483 , piece_id=H5JUJ4B54KE57CX3AIQENSZ5TNZGEL4KWDIQD5X4AEZLMXG3ORGQ, seg_pos={0 0}, upload=483852]: Long tail cancellation due to optimal threshold Dec 28 15:20:43.282235757 (551h51m43.153318792s) [noise=true , address=77.90.31.16:4736 , piece_id=UR34NNCT6QSWZFG7VIMQREBEJM4FAO7U5W2C354XGBDLZJHNZ7YA, seg_pos={0 0}, upload=483852]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk): protocol: expected piece hash; context canceled; EOF Dec 28 15:20:43.282252710 (551h51m43.153335745s) [noise=true , address=77.90.31.16:4736 , piece_id=UR34NNCT6QSWZFG7VIMQREBEJM4FAO7U5W2C354XGBDLZJHNZ7YA, seg_pos={0 0}, upload=483852]: Long tail cancellation due to optimal threshold Dec 28 15:20:43.282317734 (551h51m43.153400769s) [noise=true , address=145.79.227.44:4193 , piece_id=M2XNJ2OSDTN23RNRZ4X5QI7VTFKEL7ZKSY4SWKTLK7ZU3BW2FE3Q, seg_pos={0 0}, upload=483852]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 28 15:20:43.282325318 (551h51m43.153408363s) [noise=true , address=145.79.227.44:4193 , piece_id=M2XNJ2OSDTN23RNRZ4X5QI7VTFKEL7ZKSY4SWKTLK7ZU3BW2FE3Q, seg_pos={0 0}, upload=483852]: Long tail cancellation due to optimal threshold Dec 28 15:20:43.282367418 (551h51m43.153450453s) [seg_pos={0 0}, upload=483852]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 15:20:43.282497877 (551h51m43.153580912s) [upload=483852]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 15:20:43.282503738 (551h51m43.153586773s) [upload=483852]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 15:20:43.579155893 (551h51m43.450238938s) [upload=483852]: Done uploading segments... Dec 28 15:20:43.579182033 (551h51m43.450265078s) [upload=483852]: Upload finished. err: <nil> Duration: 30.004014457s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 11.844223507s trace: https://jaeger.dev.storj.io/trace/70ba9ccd98ccb34e FAIL (2025-12-28 15:21:02 UTC) orbiter warning: context deadline exceeded Duration: 11.84425112s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 12.333209714s trace: https://jaeger.dev.storj.io/trace/4ce8174d0eaf2b7b FAIL (2025-12-28 15:20:58 UTC) orbiter warning: context deadline exceeded Duration: 12.333242777s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [........X......X....X..........................X....X...X...] duration: 53.233256978s trace: https://jaeger.dev.storj.io/trace/51e2216a4efa8b95 FAIL (2025-12-28 15:20:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 15s) [noise=true , address=145.79.227.40:4083 , piece_id=RSMZDEGC4MX5TFRFKI33WNDK42NZDQ4M6UPRQWO4GE6VWGCJ6AFA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.854006008 (551h48m43.725089043s) [noise=true , address=77.90.31.12:4610 , piece_id=RSI6EFWFTX4JZUBEFTV3VPMJWMWI7VDMUHG6DMK7CSKTH7MDFGXA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.854333892 (551h48m43.725416937s) [noise=true , address=64.49.13.139:4476 , piece_id=FUSCXA3KPS24G3R5ORJTDOUOU2KMIK5SNBPROHL2CRKMKBRWFN6A, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.854753131 (551h48m43.725836176s) [noise=true , address=145.79.227.34:4932 , piece_id=5NGLBPM6NRZIR4NJTA43KIFQQTD6DS6B37DIFCHXGT5YMDGNMXKA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.855102245 (551h48m43.726185381s) [noise=true , address=77.90.31.15:4727 , piece_id=YJDXXQTMZQRYCEAKLZUFPW6JO7RWHJMJAOFMURYFNLQSJ4LJG2CQ, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.855467040 (551h48m43.726550076s) [noise=true , address=145.79.227.6:4280 , piece_id=ARMYEAFMP6DPW2JSHWHOYPJ6OUUSUQETJCSSQQXFQX4DDCVZYLYQ, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.855834780 (551h48m43.726917815s) [noise=true , address=193.149.164.143:4694 , piece_id=SDK5DIHWVFOZQIHQWJMZLOTQJEKRDBUTQ7R7ZTKOKF5LGECJRMOA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.856263167 (551h48m43.727346212s) [noise=true , address=145.79.227.43:4167 , piece_id=WEXP7LDX6O7KPXA6XIMRWNTMGNGSTE3XRFODPNMWFKR3D5WXX3LQ, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.858002919 (551h48m43.729085964s) [noise=true , address=145.79.227.30:4766 , piece_id=YA4HOOMXQRQCPE3VMQ6WEXBM3ZIED4RZLVFHZHRLBQJFQBFP2F7A, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.858063876 (551h48m43.729146911s) [noise=true , address=64.49.13.169:4716 , piece_id=V3PN2MUBRFNCLCYOVHYLXUORQO4S2TLRL6HMGJQCMJ6PCBZISPKQ, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.859819067 (551h48m43.730902113s) [noise=true , address=145.79.227.49:4367 , piece_id=CITJCIKXK4LYSUV6QRTLIHB7SEH5QOPUM7XR4JDHV54EJ5FLLMDQ, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.861538431 (551h48m43.732621466s) [noise=true , address=145.79.227.33:4904 , piece_id=4B64RHS2HQ6WZYND6MZGU4AZC4LQBXPNKOE2ABNVZQTEBXALQCDA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.863179095 (551h48m43.73426214s) [noise=true , address=77.90.31.19:4874 , piece_id=IXOOASSSDYJS42HWYZKK4PKE4LW47BT4WF5WRRXY7ANPLY632LRA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.864788100 (551h48m43.735871145s) [noise=true , address=77.90.31.16:4749 , piece_id=IVHKFK76TGLMJ7IBVS5ZCCMB35LKHF4FE5HLE7P5QAVGAWGSRHNA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.866240675 (551h48m43.73732371s) [noise=true , address=45.91.116.7:4029 , piece_id=VTKGXA2SWJZHOLLWREHQO66OHQZWHVRK25STFR62PEIJFGD4SG4A, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.867523768 (551h48m43.738606803s) [noise=true , address=45.91.116.6:4998 , piece_id=KCEYZDX2WGV7UCYX4H4BNSR2CWOHGPSEW76LKU2BGC3UDHA5M5SA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:45.143085883 (551h48m45.014168928s) [upload=483809]: Next returned nil segment Dec 28 15:17:45.143099779 (551h48m45.014182824s) [upload=483809]: Waiting for error group managing segments... Dec 28 15:17:45.472957433 (551h48m45.344040468s) [noise=true , address=64.49.13.139:4476 , piece_id=FUSCXA3KPS24G3R5ORJTDOUOU2KMIK5SNBPROHL2CRKMKBRWFN6A, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:45.739300305 (551h48m45.61038334s) [noise=true , address=145.79.227.6:4280 , piece_id=ARMYEAFMP6DPW2JSHWHOYPJ6OUUSUQETJCSSQQXFQX4DDCVZYLYQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:45.743884264 (551h48m45.614967309s) [noise=true , address=145.79.227.21:4508 , piece_id=PGAOV6UBGJPU7M5L3IR2LQU5LSISUJVKKBB7ZRV233F5MBEAWK6A, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:45.936004292 (551h48m45.807087328s) [noise=true , address=77.90.31.15:4727 , piece_id=YJDXXQTMZQRYCEAKLZUFPW6JO7RWHJMJAOFMURYFNLQSJ4LJG2CQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:45.968797306 (551h48m45.839880351s) [noise=true , address=77.90.31.16:4749 , piece_id=IVHKFK76TGLMJ7IBVS5ZCCMB35LKHF4FE5HLE7P5QAVGAWGSRHNA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.007515762 (551h48m45.878598797s) [noise=true , address=77.90.31.11:4580 , piece_id=UUPDJ75SL3HOJREC2SXJGQMUVLLP5WLN2L3LEON6KIKPB24DEKPQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.147397009 (551h48m46.018480054s) [noise=true , address=45.91.116.7:4029 , piece_id=VTKGXA2SWJZHOLLWREHQO66OHQZWHVRK25STFR62PEIJFGD4SG4A, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.460756007 (551h48m46.331839042s) [noise=true , address=193.149.164.143:4694 , piece_id=SDK5DIHWVFOZQIHQWJMZLOTQJEKRDBUTQ7R7ZTKOKF5LGECJRMOA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.662946081 (551h48m46.534029127s) [noise=true , address=145.79.227.43:4167 , piece_id=WEXP7LDX6O7KPXA6XIMRWNTMGNGSTE3XRFODPNMWFKR3D5WXX3LQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.695276604 (551h48m46.566359639s) [noise=true , address=45.91.116.6:4998 , piece_id=KCEYZDX2WGV7UCYX4H4BNSR2CWOHGPSEW76LKU2BGC3UDHA5M5SA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.739405984 (551h48m46.610489029s) [noise=true , address=145.79.227.33:4904 , piece_id=4B64RHS2HQ6WZYND6MZGU4AZC4LQBXPNKOE2ABNVZQTEBXALQCDA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.764293434 (551h48m46.635376479s) [noise=true , address=145.79.227.40:4083 , piece_id=RSMZDEGC4MX5TFRFKI33WNDK42NZDQ4M6UPRQWO4GE6VWGCJ6AFA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.771299524 (551h48m46.64238256s) [noise=true , address=145.79.227.30:4766 , piece_id=YA4HOOMXQRQCPE3VMQ6WEXBM3ZIED4RZLVFHZHRLBQJFQBFP2F7A, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.791263379 (551h48m46.662346414s) [noise=true , address=77.90.31.19:4874 , piece_id=IXOOASSSDYJS42HWYZKK4PKE4LW47BT4WF5WRRXY7ANPLY632LRA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.791282065 (551h48m46.6623651s) [seg_pos={0 0}, upload=483809]: Segment reached optimal threshold of 14 pieces. Dec 28 15:17:46.791451286 (551h48m46.662534321s) [noise=true , address=145.79.227.49:4367 , piece_id=CITJCIKXK4LYSUV6QRTLIHB7SEH5QOPUM7XR4JDHV54EJ5FLLMDQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF Dec 28 15:17:46.791458450 (551h48m46.662541496s) [noise=true , address=145.79.227.49:4367 , piece_id=CITJCIKXK4LYSUV6QRTLIHB7SEH5QOPUM7XR4JDHV54EJ5FLLMDQ, seg_pos={0 0}, upload=483809]: Long tail cancellation due to optimal threshold Dec 28 15:17:46.791491023 (551h48m46.662574058s) [noise=true , address=77.90.31.12:4610 , piece_id=RSI6EFWFTX4JZUBEFTV3VPMJWMWI7VDMUHG6DMK7CSKTH7MDFGXA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi): protocol: expected piece hash; context canceled; EOF Dec 28 15:17:46.791502935 (551h48m46.66258598s) [noise=true , address=77.90.31.12:4610 , piece_id=RSI6EFWFTX4JZUBEFTV3VPMJWMWI7VDMUHG6DMK7CSKTH7MDFGXA, seg_pos={0 0}, upload=483809]: Long tail cancellation due to optimal threshold Dec 28 15:17:46.791556076 (551h48m46.662639111s) [noise=true , address=145.79.227.34:4932 , piece_id=5NGLBPM6NRZIR4NJTA43KIFQQTD6DS6B37DIFCHXGT5YMDGNMXKA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qJfPZErJx4XGBw7NmfCLS7BogRnN29rUibuWkzQDFUNvPBpmm): protocol: expected piece hash; context canceled; EOF Dec 28 15:17:46.791561086 (551h48m46.662644131s) [noise=true , address=145.79.227.34:4932 , piece_id=5NGLBPM6NRZIR4NJTA43KIFQQTD6DS6B37DIFCHXGT5YMDGNMXKA, seg_pos={0 0}, upload=483809]: Long tail cancellation due to optimal threshold Dec 28 15:17:46.791659873 (551h48m46.662742908s) [noise=true , address=64.49.13.169:4716 , piece_id=V3PN2MUBRFNCLCYOVHYLXUORQO4S2TLRL6HMGJQCMJ6PCBZISPKQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:48986->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:48986->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48986->64.49.13.169:4716: use of closed network connection Dec 28 15:17:46.791665625 (551h48m46.66274866s) [noise=true , address=64.49.13.169:4716 , piece_id=V3PN2MUBRFNCLCYOVHYLXUORQO4S2TLRL6HMGJQCMJ6PCBZISPKQ, seg_pos={0 0}, upload=483809]: Long tail cancellation due to optimal threshold Dec 28 15:17:46.791739836 (551h48m46.662822881s) [seg_pos={0 0}, upload=483809]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 15:17:46.791808337 (551h48m46.662891372s) [upload=483809]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 15:17:46.791812865 (551h48m46.6628959s) [upload=483809]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 15:17:47.097154206 (551h48m46.968237251s) [upload=483809]: Done uploading segments... Dec 28 15:17:47.097173253 (551h48m46.968256288s) [upload=483809]: Upload finished. err: <nil> Duration: 53.233276025s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 42 out of 60 is failed (threshold=6) [XXXXXXXX.X..XXXXX.X.X.XXXXX...X.X.XX.XXX.X.XXXXX.XX.XX.X.XXX] duration: 5.001267499s trace: https://jaeger.dev.storj.io/trace/4224e850e2f8896a FAIL (2025-12-28 15:20:47 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001287767s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [..X...X...X.X.........X.....X......X.X.X.....X......XX.X..XX] duration: 30.003535735s trace: https://jaeger.dev.storj.io/trace/db15aef20bdee16 FAIL (2025-12-28 15:20:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 28 15:20:06.073324022 (551h51m5.944407067s) [upload=483844]: Starting upload Dec 28 15:20:06.073471593 (551h51m5.944554638s) [upload=483844]: Done starting upload Dec 28 15:20:06.073583406 (551h51m5.944666451s) [upload=483844]: Uploading segments... Dec 28 15:20:06.073713213 (551h51m5.944796258s) [upload=483844]: Got next segment. Inline: false Dec 28 15:20:06.073854292 (551h51m5.944937337s) [upload=483844]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 28 15:20:06.073862307 (551h51m5.944945342s) [upload=483844]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 28 15:20:06.318131136 (551h51m6.189214181s) [seg_pos={0 0}, upload=483844]: Begin upload segment... Dec 28 15:20:06.318187433 (551h51m6.189270478s) [seg_pos={0 0}, upload=483844]: Done begin upload segment. Dec 28 15:20:06.318222790 (551h51m6.189305836s) [noise=true , address=145.79.227.41:4086 , piece_id=E2YROMJ7QU54RVHLXHW52JIS7Q7H7OLKDGX3V3PHQUYMFGJHZYGA, seg_pos={0 0}, upload=483844]: Uploading piece... Dec 28 15:20:06.318253398 (551h51m6.189336443s) [noise=true , address=145.79.227.33:4870 , piece_id=OHXSBLNPJYMT2UQDUQ3U6SGR7LPVOKI5IG4AHERG2TO7HENQT5OQ, seg_pos={0 0}, upload=483844]: Uploading piece... Dec 28 15:20:06.319193860 (551h51m6.190276905s) [noise=true , address=145.79.227.13:4802 , piece_id=4N3HHWOB6LWYOZW466OQ2BR4AD3NNZCBDZ44E53VEHEADW3G7WOQ, seg_pos={0 0}, upload=483844]: Uploading piece... Dec 28 15:20:06.319614450 (551h51m6.190697496s) [noise=true , address=145.79.227.15:4874 , piece_id=VRQ4GX4URKJAHOWQ2PN7NPQYWHT6DBCGIGBXHK2NEDFUHHBMN2CA, seg_pos={0 0}, upload=483844]: Uploading piece... Dec 28 15:20:06.320049599 (551h51m6.191132644s) [noise=true , address=145.79.227.16:4911 , piece_id=YMULANGCVGIBVWVONDV5HDYRK2CKVIBKOHGUY3N44H5HD7ZRT2RQ, seg_pos={0 0}, upload=483844]: Uploading piece... Dec 28 15:20:06.320526587 (551h51m6.191609632s) [noise=true , address=77.90.31.11:4553 , piece_id=OHVKBLNVOF2THGZDGZPP5PX6GENXC4T7TBY3JZ7GXCPS3RG2UAGQ, seg_pos={0 0}, upload=483844]: Uploading piece... Dec 28 15:20:06.320961485 (551h51m6.19204453s) [noise=true , address=77.90.31.14:4663 , piece_id=COL6LY322P57B6LCATMNTDRKJRPF33QNKW6IGX7EST3TUS32GEMA, seg_pos={0 0}, upload=483844]: Uploading piece... Dec 28 15:20:06.321402425 (551h51m6.19248547s) [noise=true , address=45.91.116.10:4084 , piece_id=TDGK7F4KJQSN37XZH4UZG6MJFFEIYRRZZNRZ7BUWA4PPIQYFOEIQ, seg_pos={0 0}, upload=483844]: Uploading piece... Dec 28 15:20:06.322020903 (551h51m6.193103948s) [noise=true , address=145.79.227.38:4978 , piece_id=CM2GM27BEVERUS4OA6FVPZT22IQZLCX7RW2JIIG5PQHLOT3SNPUQ, seg_pos={0 0}, upload=483844]: Uploading piece... Dec 28 15:20:06.324000853 (551h51m6.195083898s) [noise=true , address=145.79.227.47:4265 , piece_id=XXDJ7LBXBCHVXSUEVL2ZQDS372N5PKCS7CBTGSSF7A3RFSSLZHZQ, seg_pos={0 0}, upload=483844]: Uploading piece... Dec 28 15:20:06.325941217 (551h51m6.197024272s) [noise=true , address=145.79.227.8:4330 , piece_id=QCFAKMF3QUWBNLW6U5G7PXASIFBE24QYJEBAJ4PT5VJ4WT5RZSXA, seg_pos={0 0}, upload=483844]: Uploading piece... Dec 28 15:20:06.327968097 (551h51m6.199051142s) [noise=true , address=145.79.227.46:4603 , piece_id=QDLFLIUSXRB7ZLYMR53YCAPQSLV7S7JDGOOM3YGLY7DC5SDXU55A, seg_pos={0 0}, upload=483844]: Uploading piece... Dec 28 15:20:06.329952035 (551h51m6.20103508s) [noise=true , address=77.90.31.18:4808 , piece_id=KRIY5L2XT6NYEFLNTT76MPOOTHCBNNFLL2BFQWDRKJ7LNKHXG2GA, seg_pos={0 0}, upload=483844]: Uploading piece... Dec 28 15:20:06.331888502 (551h51m6.202971547s) [noise=true , address=45.91.116.13:4156 , piece_id=QZNEC5G2LLPB5AEO2JXAQM3HLVUUKCG64BRUOIYTPJYYUGWOFROQ, seg_pos={0 0}, upload=483844]: Uploading piece... Dec 28 15:20:06.333199739 (551h51m6.204282784s) [noise=true , address=145.79.227.52:4445 , piece_id=ATSEI6XTJXNFDTMD7ZOSBK6NHMS4ELA5QPWGGSGV3IYIGE2RJANQ, seg_pos={0 0}, upload=483844]: Uploading piece... Dec 28 15:20:06.337398544 (551h51m6.208481589s) [noise=true , address=145.79.227.5:4221 , piece_id=X37LGV7TFBCVSZSUSHOOS5KX3ZZKYA45UXUEX7W3FRRHAXBTM2JQ, seg_pos={0 0}, upload=483844]: Uploading piece... Dec 28 15:20:08.031244049 (551h51m7.902327094s) [upload=483844]: Next returned nil segment Dec 28 15:20:08.031261582 (551h51m7.902344617s) [upload=483844]: Waiting for error group managing segments... Dec 28 15:20:08.322924177 (551h51m8.194007212s) [noise=true , address=145.79.227.38:4978 , piece_id=CM2GM27BEVERUS4OA6FVPZT22IQZLCX7RW2JIIG5PQHLOT3SNPUQ, seg_pos={0 0}, upload=483844]: Done uploading piece. err: <nil> Dec 28 15:20:08.328092729 (551h51m8.199175774s) [noise=true , address=145.79.227.15:4874 , piece_id=VRQ4GX4URKJAHOWQ2PN7NPQYWHT6DBCGIGBXHK2NEDFUHHBMN2CA, seg_pos={0 0}, upload=483844]: Done uploading piece. err: <nil> Dec 28 15:20:08.333126534 (551h51m8.204209579s) [noise=true , address=145.79.227.16:4911 , piece_id=YMULANGCVGIBVWVONDV5HDYRK2CKVIBKOHGUY3N44H5HD7ZRT2RQ, seg_pos={0 0}, upload=483844]: Done uploading piece. err: <nil> Dec 28 15:20:08.343859311 (551h51m8.214942366s) [noise=true , address=145.79.227.13:4802 , piece_id=4N3HHWOB6LWYOZW466OQ2BR4AD3NNZCBDZ44E53VEHEADW3G7WOQ, seg_pos={0 0}, upload=483844]: Done uploading piece. err: <nil> Dec 28 15:20:08.345743829 (551h51m8.216826875s) [noise=true , address=145.79.227.46:4603 , piece_id=QDLFLIUSXRB7ZLYMR53YCAPQSLV7S7JDGOOM3YGLY7DC5SDXU55A, seg_pos={0 0}, upload=483844]: Done uploading piece. err: <nil> Dec 28 15:20:08.346303956 (551h51m8.217387001s) [noise=true , address=145.79.227.8:4330 , piece_id=QCFAKMF3QUWBNLW6U5G7PXASIFBE24QYJEBAJ4PT5VJ4WT5RZSXA, seg_pos={0 0}, upload=483844]: Done uploading piece. err: <nil> Dec 28 15:20:08.356093437 (551h51m8.227176482s) [noise=true , address=145.79.227.5:4221 , piece_id=X37LGV7TFBCVSZSUSHOOS5KX3ZZKYA45UXUEX7W3FRRHAXBTM2JQ, seg_pos={0 0}, upload=483844]: Done uploading piece. err: <nil> Dec 28 15:20:08.367173384 (551h51m8.238256429s) [noise=true , address=145.79.227.52:4445 , piece_id=ATSEI6XTJXNFDTMD7ZOSBK6NHMS4ELA5QPWGGSGV3IYIGE2RJANQ, seg_pos={0 0}, upload=483844]: Done uploading piece. err: <nil> Dec 28 15:20:08.367533300 (551h51m8.238616345s) [noise=true , address=145.79.227.41:4086 , piece_id=E2YROMJ7QU54RVHLXHW52JIS7Q7H7OLKDGX3V3PHQUYMFGJHZYGA, seg_pos={0 0}, upload=483844]: Done uploading piece. err: <nil> Dec 28 15:20:08.375573621 (551h51m8.246656666s) [noise=true , address=145.79.227.47:4265 , piece_id=XXDJ7LBXBCHVXSUEVL2ZQDS372N5PKCS7CBTGSSF7A3RFSSLZHZQ, seg_pos={0 0}, upload=483844]: Done uploading piece. err: <nil> Dec 28 15:20:08.581994329 (551h51m8.453077375s) [noise=true , address=45.91.116.13:4156 , piece_id=QZNEC5G2LLPB5AEO2JXAQM3HLVUUKCG64BRUOIYTPJYYUGWOFROQ, seg_pos={0 0}, upload=483844]: Done uploading piece. err: <nil> Dec 28 15:20:08.611146771 (551h51m8.482229806s) [noise=true , address=45.91.116.10:4084 , piece_id=TDGK7F4KJQSN37XZH4UZG6MJFFEIYRRZZNRZ7BUWA4PPIQYFOEIQ, seg_pos={0 0}, upload=483844]: Done uploading piece. err: <nil> Dec 28 15:20:08.621718200 (551h51m8.492801236s) [noise=true , address=77.90.31.11:4553 , piece_id=OHVKBLNVOF2THGZDGZPP5PX6GENXC4T7TBY3JZ7GXCPS3RG2UAGQ, seg_pos={0 0}, upload=483844]: Done uploading piece. err: <nil> Dec 28 15:20:08.621748027 (551h51m8.492831072s) [seg_pos={0 0}, upload=483844]: Segment reached optimal threshold of 13 pieces. Dec 28 15:20:08.622065662 (551h51m8.493148707s) [noise=true , address=77.90.31.14:4663 , piece_id=COL6LY322P57B6LCATMNTDRKJRPF33QNKW6IGX7EST3TUS32GEMA, seg_pos={0 0}, upload=483844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:155F5vnBUAgJYDuEgxwgnHCEVarkUAcdNjqh7V52vx65uHyJfE): protocol: expected piece hash; context canceled; EOF Dec 28 15:20:08.622077274 (551h51m8.493160309s) [noise=true , address=77.90.31.14:4663 , piece_id=COL6LY322P57B6LCATMNTDRKJRPF33QNKW6IGX7EST3TUS32GEMA, seg_pos={0 0}, upload=483844]: Long tail cancellation due to optimal threshold Dec 28 15:20:08.622105868 (551h51m8.493188914s) [noise=true , address=145.79.227.33:4870 , piece_id=OHXSBLNPJYMT2UQDUQ3U6SGR7LPVOKI5IG4AHERG2TO7HENQT5OQ, seg_pos={0 0}, upload=483844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sDH7ccMQhjePqpbTkkxMAtumNLxpMHiT56cnfaYrUTFYxuPVt): protocol: expected piece hash; context canceled; EOF Dec 28 15:20:08.622115097 (551h51m8.493198142s) [noise=true , address=145.79.227.33:4870 , piece_id=OHXSBLNPJYMT2UQDUQ3U6SGR7LPVOKI5IG4AHERG2TO7HENQT5OQ, seg_pos={0 0}, upload=483844]: Long tail cancellation due to optimal threshold Dec 28 15:20:08.622294368 (551h51m8.493377423s) [noise=true , address=77.90.31.18:4808 , piece_id=KRIY5L2XT6NYEFLNTT76MPOOTHCBNNFLL2BFQWDRKJ7LNKHXG2GA, seg_pos={0 0}, upload=483844]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9): protocol: expected piece hash; context canceled; EOF Dec 28 15:20:08.622302363 (551h51m8.493385398s) [noise=true , address=77.90.31.18:4808 , piece_id=KRIY5L2XT6NYEFLNTT76MPOOTHCBNNFLL2BFQWDRKJ7LNKHXG2GA, seg_pos={0 0}, upload=483844]: Long tail cancellation due to optimal threshold Dec 28 15:20:08.622321809 (551h51m8.493404865s) [seg_pos={0 0}, upload=483844]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 28 15:20:08.622407663 (551h51m8.493490709s) [upload=483844]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 15:20:08.622507263 (551h51m8.493590298s) [upload=483844]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 15:20:08.917013954 (551h51m8.788097009s) [upload=483844]: Done uploading segments... Dec 28 15:20:08.917038860 (551h51m8.788121905s) [upload=483844]: Upload finished. err: <nil> Duration: 30.003567895s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 6 out of 60 is failed (threshold=3) [.........X......X....X..........................X....X...X..] duration: 41.753394539s trace: https://jaeger.dev.storj.io/trace/51e2216a4efa8b95 FAIL (2025-12-28 15:20:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 15s) [noise=true , address=145.79.227.40:4083 , piece_id=RSMZDEGC4MX5TFRFKI33WNDK42NZDQ4M6UPRQWO4GE6VWGCJ6AFA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.854006008 (551h48m43.725089043s) [noise=true , address=77.90.31.12:4610 , piece_id=RSI6EFWFTX4JZUBEFTV3VPMJWMWI7VDMUHG6DMK7CSKTH7MDFGXA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.854333892 (551h48m43.725416937s) [noise=true , address=64.49.13.139:4476 , piece_id=FUSCXA3KPS24G3R5ORJTDOUOU2KMIK5SNBPROHL2CRKMKBRWFN6A, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.854753131 (551h48m43.725836176s) [noise=true , address=145.79.227.34:4932 , piece_id=5NGLBPM6NRZIR4NJTA43KIFQQTD6DS6B37DIFCHXGT5YMDGNMXKA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.855102245 (551h48m43.726185381s) [noise=true , address=77.90.31.15:4727 , piece_id=YJDXXQTMZQRYCEAKLZUFPW6JO7RWHJMJAOFMURYFNLQSJ4LJG2CQ, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.855467040 (551h48m43.726550076s) [noise=true , address=145.79.227.6:4280 , piece_id=ARMYEAFMP6DPW2JSHWHOYPJ6OUUSUQETJCSSQQXFQX4DDCVZYLYQ, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.855834780 (551h48m43.726917815s) [noise=true , address=193.149.164.143:4694 , piece_id=SDK5DIHWVFOZQIHQWJMZLOTQJEKRDBUTQ7R7ZTKOKF5LGECJRMOA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.856263167 (551h48m43.727346212s) [noise=true , address=145.79.227.43:4167 , piece_id=WEXP7LDX6O7KPXA6XIMRWNTMGNGSTE3XRFODPNMWFKR3D5WXX3LQ, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.858002919 (551h48m43.729085964s) [noise=true , address=145.79.227.30:4766 , piece_id=YA4HOOMXQRQCPE3VMQ6WEXBM3ZIED4RZLVFHZHRLBQJFQBFP2F7A, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.858063876 (551h48m43.729146911s) [noise=true , address=64.49.13.169:4716 , piece_id=V3PN2MUBRFNCLCYOVHYLXUORQO4S2TLRL6HMGJQCMJ6PCBZISPKQ, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.859819067 (551h48m43.730902113s) [noise=true , address=145.79.227.49:4367 , piece_id=CITJCIKXK4LYSUV6QRTLIHB7SEH5QOPUM7XR4JDHV54EJ5FLLMDQ, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.861538431 (551h48m43.732621466s) [noise=true , address=145.79.227.33:4904 , piece_id=4B64RHS2HQ6WZYND6MZGU4AZC4LQBXPNKOE2ABNVZQTEBXALQCDA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.863179095 (551h48m43.73426214s) [noise=true , address=77.90.31.19:4874 , piece_id=IXOOASSSDYJS42HWYZKK4PKE4LW47BT4WF5WRRXY7ANPLY632LRA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.864788100 (551h48m43.735871145s) [noise=true , address=77.90.31.16:4749 , piece_id=IVHKFK76TGLMJ7IBVS5ZCCMB35LKHF4FE5HLE7P5QAVGAWGSRHNA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.866240675 (551h48m43.73732371s) [noise=true , address=45.91.116.7:4029 , piece_id=VTKGXA2SWJZHOLLWREHQO66OHQZWHVRK25STFR62PEIJFGD4SG4A, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:43.867523768 (551h48m43.738606803s) [noise=true , address=45.91.116.6:4998 , piece_id=KCEYZDX2WGV7UCYX4H4BNSR2CWOHGPSEW76LKU2BGC3UDHA5M5SA, seg_pos={0 0}, upload=483809]: Uploading piece... Dec 28 15:17:45.143085883 (551h48m45.014168928s) [upload=483809]: Next returned nil segment Dec 28 15:17:45.143099779 (551h48m45.014182824s) [upload=483809]: Waiting for error group managing segments... Dec 28 15:17:45.472957433 (551h48m45.344040468s) [noise=true , address=64.49.13.139:4476 , piece_id=FUSCXA3KPS24G3R5ORJTDOUOU2KMIK5SNBPROHL2CRKMKBRWFN6A, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:45.739300305 (551h48m45.61038334s) [noise=true , address=145.79.227.6:4280 , piece_id=ARMYEAFMP6DPW2JSHWHOYPJ6OUUSUQETJCSSQQXFQX4DDCVZYLYQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:45.743884264 (551h48m45.614967309s) [noise=true , address=145.79.227.21:4508 , piece_id=PGAOV6UBGJPU7M5L3IR2LQU5LSISUJVKKBB7ZRV233F5MBEAWK6A, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:45.936004292 (551h48m45.807087328s) [noise=true , address=77.90.31.15:4727 , piece_id=YJDXXQTMZQRYCEAKLZUFPW6JO7RWHJMJAOFMURYFNLQSJ4LJG2CQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:45.968797306 (551h48m45.839880351s) [noise=true , address=77.90.31.16:4749 , piece_id=IVHKFK76TGLMJ7IBVS5ZCCMB35LKHF4FE5HLE7P5QAVGAWGSRHNA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.007515762 (551h48m45.878598797s) [noise=true , address=77.90.31.11:4580 , piece_id=UUPDJ75SL3HOJREC2SXJGQMUVLLP5WLN2L3LEON6KIKPB24DEKPQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.147397009 (551h48m46.018480054s) [noise=true , address=45.91.116.7:4029 , piece_id=VTKGXA2SWJZHOLLWREHQO66OHQZWHVRK25STFR62PEIJFGD4SG4A, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.460756007 (551h48m46.331839042s) [noise=true , address=193.149.164.143:4694 , piece_id=SDK5DIHWVFOZQIHQWJMZLOTQJEKRDBUTQ7R7ZTKOKF5LGECJRMOA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.662946081 (551h48m46.534029127s) [noise=true , address=145.79.227.43:4167 , piece_id=WEXP7LDX6O7KPXA6XIMRWNTMGNGSTE3XRFODPNMWFKR3D5WXX3LQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.695276604 (551h48m46.566359639s) [noise=true , address=45.91.116.6:4998 , piece_id=KCEYZDX2WGV7UCYX4H4BNSR2CWOHGPSEW76LKU2BGC3UDHA5M5SA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.739405984 (551h48m46.610489029s) [noise=true , address=145.79.227.33:4904 , piece_id=4B64RHS2HQ6WZYND6MZGU4AZC4LQBXPNKOE2ABNVZQTEBXALQCDA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.764293434 (551h48m46.635376479s) [noise=true , address=145.79.227.40:4083 , piece_id=RSMZDEGC4MX5TFRFKI33WNDK42NZDQ4M6UPRQWO4GE6VWGCJ6AFA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.771299524 (551h48m46.64238256s) [noise=true , address=145.79.227.30:4766 , piece_id=YA4HOOMXQRQCPE3VMQ6WEXBM3ZIED4RZLVFHZHRLBQJFQBFP2F7A, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.791263379 (551h48m46.662346414s) [noise=true , address=77.90.31.19:4874 , piece_id=IXOOASSSDYJS42HWYZKK4PKE4LW47BT4WF5WRRXY7ANPLY632LRA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: <nil> Dec 28 15:17:46.791282065 (551h48m46.6623651s) [seg_pos={0 0}, upload=483809]: Segment reached optimal threshold of 14 pieces. Dec 28 15:17:46.791451286 (551h48m46.662534321s) [noise=true , address=145.79.227.49:4367 , piece_id=CITJCIKXK4LYSUV6QRTLIHB7SEH5QOPUM7XR4JDHV54EJ5FLLMDQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12C4Xom8WT5TZRiX3ZFyRT9tSoAUUFyPNS1VVzN7gT3oDVGDGhK): protocol: expected piece hash; context canceled; EOF Dec 28 15:17:46.791458450 (551h48m46.662541496s) [noise=true , address=145.79.227.49:4367 , piece_id=CITJCIKXK4LYSUV6QRTLIHB7SEH5QOPUM7XR4JDHV54EJ5FLLMDQ, seg_pos={0 0}, upload=483809]: Long tail cancellation due to optimal threshold Dec 28 15:17:46.791491023 (551h48m46.662574058s) [noise=true , address=77.90.31.12:4610 , piece_id=RSI6EFWFTX4JZUBEFTV3VPMJWMWI7VDMUHG6DMK7CSKTH7MDFGXA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12aJUUvWNCYSbhTkNgjAj6FPXK5LK2qdHDqntvrVabGwD2BhsYi): protocol: expected piece hash; context canceled; EOF Dec 28 15:17:46.791502935 (551h48m46.66258598s) [noise=true , address=77.90.31.12:4610 , piece_id=RSI6EFWFTX4JZUBEFTV3VPMJWMWI7VDMUHG6DMK7CSKTH7MDFGXA, seg_pos={0 0}, upload=483809]: Long tail cancellation due to optimal threshold Dec 28 15:17:46.791556076 (551h48m46.662639111s) [noise=true , address=145.79.227.34:4932 , piece_id=5NGLBPM6NRZIR4NJTA43KIFQQTD6DS6B37DIFCHXGT5YMDGNMXKA, seg_pos={0 0}, upload=483809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1qJfPZErJx4XGBw7NmfCLS7BogRnN29rUibuWkzQDFUNvPBpmm): protocol: expected piece hash; context canceled; EOF Dec 28 15:17:46.791561086 (551h48m46.662644131s) [noise=true , address=145.79.227.34:4932 , piece_id=5NGLBPM6NRZIR4NJTA43KIFQQTD6DS6B37DIFCHXGT5YMDGNMXKA, seg_pos={0 0}, upload=483809]: Long tail cancellation due to optimal threshold Dec 28 15:17:46.791659873 (551h48m46.662742908s) [noise=true , address=64.49.13.169:4716 , piece_id=V3PN2MUBRFNCLCYOVHYLXUORQO4S2TLRL6HMGJQCMJ6PCBZISPKQ, seg_pos={0 0}, upload=483809]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1w21xeA7emhjphgYw2Qyz3uuHgPezuivmVD9KmrKmNZk3TnaGN): protocol: write tcp 188.166.225.170:48986->64.49.13.169:4716: use of closed network connection; write tcp 188.166.225.170:48986->64.49.13.169:4716: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:48986->64.49.13.169:4716: use of closed network connection Dec 28 15:17:46.791665625 (551h48m46.66274866s) [noise=true , address=64.49.13.169:4716 , piece_id=V3PN2MUBRFNCLCYOVHYLXUORQO4S2TLRL6HMGJQCMJ6PCBZISPKQ, seg_pos={0 0}, upload=483809]: Long tail cancellation due to optimal threshold Dec 28 15:17:46.791739836 (551h48m46.662822881s) [seg_pos={0 0}, upload=483809]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil> Dec 28 15:17:46.791808337 (551h48m46.662891372s) [upload=483809]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 28 15:17:46.791812865 (551h48m46.6628959s) [upload=483809]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 28 15:17:47.097154206 (551h48m46.968237251s) [upload=483809]: Done uploading segments... Dec 28 15:17:47.097173253 (551h48m46.968256288s) [upload=483809]: Upload finished. err: <nil> Duration: 41.753418957s core uplink us-select-1 us1-placement-20

1
...
11324
11325
11326
11327
11328
...
18019